Building a Fast Wikipedia Offline Reader

ttsiod writes "An internet connection is not always at hand. I wanted to install Wikipedia on my laptop to be able to carry it along with me on business trips. After trying and rejecting the normal (MySQL-based) procedure, I quickly hacked a much better one over the weekend, using open source tools. Highlights: (1) Very fast searching. (2) Keyword (actually, title words) based searching. (3) Search produces multiple possible articles, sorted by probability (you choose amongst them). (4) LaTeX based rendering for mathematical equations. (5) Hard disk usage is minimal: space for the original .bz2 file plus the index built through Xapian. (6) Orders of magnitude faster to install (a matter of hours) compared to loading the 'dump' into MySQL — which, if you want to enable keyword searching, takes days."

Wow!

[ferrocene (203243)]

After doing all that, I think you may have missed your flight! :)

Re:

[bn557 (183935)]

This may seem like a stupid, trivial, and pointless project, but the programmer may have gained something from it that he could use later in something you don't feel that way about. If the programmer enjoyed doing it, that might have lead to a more productive coding session later in the day too.

Re:Why?

[rabblerabble (884373)]

I'll bite...Unfortunately, I don't have a basement, so therefore there are times that I am required to venture into the outer realm that happens to be heated by the big ball of gas known as Sol, as opposed to a pump ;P Seriously though, this is exactly what I have been looking for. What better way to show up your friends when they cry "You're wrong, google it!" knowing that there is no connection possible within twenty miles. Next time i'm drunk at the beach and someone wants to pretend to know the history of coffee harvesting, it's on.

Just settle it the old way

[EmbeddedJanitor (597831)]

Kick sand in their face!

Re:

[rabblerabble (884373)]

The goggles would work then. Your logic is flawed.

Take that, Mr Obviously A. Troll!

[ampathee (682788)]

Programmers shouldn't be wasting time on these trivial, pointless projects. We need their work in other more important projects!

Hah! I'm going to start work on (let's see..) a random lolcat generator now, just to piss you off.

Re:

[SoapDish (971052)]

Make sure to write it in LOL-CODE! (http://lolcode.com/ [lolcode.com])

Re:Take that, Mr Obviously A. Troll!

[MarkRose (820682)]

You mean something like lolcatgenerator.com [lolcatgenerator.com]? Looks like someone already tackled that important project! lol

Re:

[RuBLed (995686)]

Programmers shouldn't be wasting time on these trivial, pointless projects. We need their work in other more important projects!

Ironically, You're already reading slashdot. You had just wasted your time.















You're reading it again, wasting more time ehh??....

But the point is, if programming an offline wikipedia makes you happy and you don't need the money then you would understand....

Re:Why?

[thePsychologist (1062886)]

Realize that some of the greatest things done by humankind were from doing "pointless projects" as you call them. Prime numbers for instance were studied by mathematicians just for fun, and now look, they're used for cryptography. Try doing your banking without them.

Complex numbers originated from something "useless" like trying to solve the quartic polynomial in radicals...try building a bridge without them. In fact all of science is built upon people going in random tangents doing things they enjoy, discovering seemingly "useless facts" but most of it becomes useful *and* gives us an idea of the universe in which we live.

Only working on immediate practical problems is very shortsighted, and if mandated throughout the academic community, would mean the death of innovation and most discoveries.

SDHC?

[tepples (727027)]

Why? Have you seen the price of 4G flash cards recently?

Yes, and it's possible that you may need a new PDA in order to use SD cards larger than 2 GB. The 4 GB ones use a different protocol called SDHC [pocketpccentral.net] that older PDAs may not support. It's analogous to the old ATA hard disk size barriers [pcguide.com], especially the 137 GB (128 GiB) barrier [pcguide.com]. Or are most PDAs capable of being upgraded to handle SDHC?

Re:

[SCHecklerX (229973)]

You can get a normal 4GB SD card from Transcend. I am using one in my sandisk sansa e140, which is definitely NOT SDHC compatible.

Ho-Hum ...

[jabberwock (10206)]

What, no auto update? No User Agreement? No disabled features that are enabled by a mammoth key? No product registration?

Let us know when you're ready for prime time ... ;-)

Re:Ho-Hum ...

[OzRoy (602691)]

Auto-update would be interesting. How do you keep the data up to date without downloading the entire 2.9G again? Is there some sort of diff file you can download?

Re:

[MichaelSmith (789609)]

How do you keep the data up to date without downloading the entire 2.9G again?

Not too hard if you have a sub-etha net connection handy. Better check that the article about The Earth which you have been working on hasn't been cut down to two words though.

I hope

[Nikron (888774)]

That you don't dump the wiki at a bad time.

George W Bush

Is a dick head!!!!11

Re:I hope

[by Anonymous Coward]

You mean before someone makes it inaccurate again?

Oh, nevermind, I see the problem:

George W Bush

Is a dick head!!!!11

should be

George W Bush

Is a dick head!!!!!!

Man, those out to mess with the content are getting more and more subtle...

But...

[by Anonymous Coward]

What's the point of it if there are no vandals or flame wars to make it interesting?

Hitchhiker's guide here we come!

[Brietech (668850)]

Combine this and one of the new E-ink ebook readers, make it pretty rugged, slap a solar panel on the back and man. . . you have something really close to a genuine hitchhiker's guide to the galaxy. Ah, I love where technology is heading =)

Re:Hitchhiker's guide here we come!

[Sneftel (15416)]

As long as hitchhikers primarily need to know how to evolve a Pikachu into a Raichu, and how Benjamin Disraeli has been referenced in pop culture.

Re:Hitchhiker's guide here we come!

[RandomWhiteMan (685768)]

You laugh now, but just wait until you're stranded in the middle of Blackheath England, needing a ride from a conservative British History Scholar who has his son with him playing Pokemon Gold. Won't be so smug then, will you. I bet you won't even have your towel on you when this all goes down.

Re:Hitchhiker's guide here we come!

[cowens (30752)]

Ah, but that is what the original HHGTTG was as well. Tons of info on alcohol and Eccentricea Gallumbits (the triple breasted whore of Eroticon Six), but the entry for Earth was: Harmless. Later it was expanded: Mostly Harmless.

Re:Hitchhiker's guide here we come!

[nstlgc (945418)]

Just so we're clear, you can make Pikachu evolve into Raichu by using the Thunder Stone (which makes sense, since they're Electric Pokémon). However, due to the emotional value Pikachu has to trainers, most of them choose not to evolve him. Some Pokémon games even plain don't allow this. I hope this was helpful.

Re:

[dch24 (904899)]

Don't forget to put this on the cover, in large reassuring letters:

DON'T PANIC

Re:Hitchhiker's guide here we come!

[Gromius (677157)]

Yes its a perfect fit. Particularly as Wikipedia has now supplanted the Encyclopedia Britannica in many places as the standard repository of all knowledge and wisdom. Although it has many omissions, contains much that is apocryphal, or at least widely inaccurate, it scores over the older more pedestrian work in two important ways.

        * 1. It is slightly cheaper
        * 2. It has the words "You can copy and edit me for free" inscribed in large friendly letters in the license.

Also like the guide, although it cannot hope to be useful or informative on all matters, it does make the reassuring claim that where it is inaccurate, it is at least definitively inaccurate :)

Only 2 days huh

[by Anonymous Coward]

I was able to build this in two days, most of which were spent searching for the appropriate tools. Simply unbelievable... toying around with these tools and writing less than 200 lines of code, and... presto!

Give that man a job at Google.

Good part of the page: the explanation

[phliar (87116)]

For a change it's not just a link to a .tar.gz somewhere, but an actual article where he goes through what he did, and (more important) why he did things that way. Good reading even if you don't want an off-line Wikipedia.

It doesn't take days

[BReflection (736785)]

It only takes days if you use the php import script to import the sql dump, which was not designed for importing the entire dump.

Use the ANSI C implementation, which takes about 20 minutes to convert the XML to SQL and then takes a few hours to import into MySQL. Please not that you need a properly configured MySQL server in order to efficiently run a local copy of Wikipedia, which must have at least 8GB of ram.

http://meta.wikimedia.org/wiki/Xml2sql [wikimedia.org]

Mass inserts into mysql...

[Splab (574204)]

is very very slow when you do it on a normal installation, the reason is MySQL comes with a "be nice to people who don't know what they are doing" setup. Go into the my.cnf and find the buffer settings, crank them up and restart the server. It can really do a lot (especially if you are running InnoDB which you of course are since MyISAM isn't a proper database).

What??

[icydog (923695)]

TFA is:

1. Not a thinly-veiled attempt to advertise a crappy product
2. Not bashing Microsoft
3. Not about somebody who is trolling open-source (i.e. SCO)
4. Not about Bush taking away all our rights and ending freedom
5. Not about voting fraud and the end of democracy/America/the world
6. Not decrying Vista DRM and its ties to the MAFIAA
7. Posted on Slashdot

Furthermore, TFA is interesting and informative.

Am I in heaven?

Re:What??

[Pollardito (781263)]

it'll get posted again tomorrow just to maintain expectations

can we get a PSP version of it?

[mu22le (766735)]

A PSP is very portable (fits in your sweater/backpack), hackable, and has up to 8Gb of storage. I have been dreaming for an year about porting wikipedia to it. Unfortunately I'm not familiar with the kind of programming needed and I could never find the time...

better yet, a DS version

[tepples (727027)]

A PSP is very portable (fits in your sweater/backpack), hackable

You have to buy a used PSP to be sure that you can hack it. New ones are likely to come with firmware version 3.51 or later, which is not cracked as of August 2007. The Nintendo DS, on the other hand, had its last major firmware update in September 2005 and is still cracked, with SLOT-1 modchips available at Wal-Mart for $30.

and has up to 8Gb of storage

So does a CompactFlash card in a GBA Movie Player in a Nintendo DS. It's a pity that the SLOT-1 adapters for DS haven't been shown to be compatible with SDHC.

There's a bug in TFA: Missing articles.

[dannycim (442761)]

There's a serious problem with the article's way of treating the data that I didn't see addressed.

The wikipedia database file is one large bzip2'ed XML file which the author splits into blocks of 900k (bzip2's natural blocking) which he then parses for the "title" and "text" XML tags.

The problem with that approach is that some of these tags may well end up being split over block boundaries, so some articles risk being missed. EG:

END-OF-BLOCK: blablablabla...blabla[/text][othertag][ti

START-OF-NEXT-BLOCK: tle][sometag]blablablablabla...

So searching for "[title]" in boths blocks separately like TFA does will fail for one article.

(I've used square brackets instead of lessthans and greaterthans because slashdot won't let me use them.)

Re:There's a bug in TFA: Missing articles.

[ttsiod (881575)]

(Re-post: for some reason the response I sent some hours ago didn't appear) No, actually there is no bug. If you read the contents of the 'show.pl' script, you'll see that it adapts to a missing '</text>' by reading from the next volume - the next recxxx...bz2.

As for the title, what you describe can't happen because of a fortunate side-effect: when compressing, bzip2 (as other compressors) look for previous appearances of a token (in this case, '<title>') and code a reference to it (instead of the full text) to save space. Since "text" and "title" appear all the time in these blocks (at least once for each article), they will NOT be split - they will be encoded as "references", and therefore, what you describe shouldn't happen (I hope :-)

Re:Uh....

[dhwebb (526291)]

Programming something new to some people is like playing a video game. I love programming useless things just for the challenge. People who don't understand that have never had a true love for programming.

Re:

[Tablizer (95088)]

I love programming useless things just for the challenge.

Have you ever worked on a project called "Clippey", by chance?
     

Re:Uh....

[Gazzonyx (982402)]

I love programming useless things just for the challenge.

Have you ever worked on a project called "Clippey", by chance?

No, he said he has a love for programming; not a seething hatred for users. Besides, everyone knows programmers only hate admins. ;) On behalf of the programmers, I'd like to say that this isn't true we love our admins. Who else makes sure that our connections*&#^$: Connection Reset By Peer

Re:Uh....

[stephanruby (542433)]

Programming something new to some people is like playing a video game.

Speaking of which, http://www.pyweek.org/ [pyweek.org] is coming up this first week of September. It's time to dust off that python book (or borrow one from someone) and do whatever you have to do to get some days off that week.

I know the feeling

[aepervius (535155)]

They say to you that their hobby is painting/music/walking/repairing old car/gardening/making reduced model etc... And they seem to think that their hobby are perfectly acceptable. But as soon as you say you like to program stuff, they don't understand how this would be a hobby. They mostly fail to recognize that every one of us has something in common : the joy of act of creation. The fact that our hobby entail creating something immaterial and full of "logic" does not matter. It is still a joy.

Re:

[fotbr (855184)]

I was that way, once. Then other hobbies came along, and now I rarely do any programming thats not work related.

Its funny how time changes you.

Re:

[hobbesmaster (592205)]

So you can settle trivial arguments with your friends when away from an internet connection, duh!

(Or to always have something to read on your laptop while traveling - this is what I would use it for)

Re:

[Gazzonyx (982402)]

So you can settle trivial arguments with your friends when away from an internet connection, duh!

(Or to always have something to read on your laptop while traveling - this is what I would use it for)

I bet you're quite the ladies man, huh?
Sorry, I couldn't resist!

Re:Just hope you don't get an effed image.

[Tacvek (948259)]

My very serious question to you is how much better do you think things are at a "real" encyclopedia. They have many of the same problems, but they are just not public. "Real" encyclopedias can be just an inaccurate as the Wikipedia on many articles. For a quick first reference, Wikipedia is an ideal tool. Just be sure to take things with a grain of salt if you are not checking the sources for further information. Guess what though, the same applies to "real" encyclopedias too. One difference is that with "real" encyclopedias, you always lack revision information, and you often lack information about the sources used by the editors. (Some encyclopedias are better than others in that respect.)

Re:Just hope you don't get an effed image.

[gad_zuki! (70830)]

Yes, the paper encyclopedias are missing all the anime trivia. Christ, its embarassing to see "references in pop culture" sections which just spell out every geeky guy stereotype. I dont know why those people dont get banned. Everything in existance has an anime reference. That is unsettling.

Re:

[ZzzzSleep (606571)]

Blatantly stolen from David Morgan-Mar [livejournal.com].

In many of the more relaxed corners of the Outer Eastern Rim of the Internet, Wikipedia has already supplanted the great Encyclopaedia Britannica as the standard repository of all knowledge and wisdom, for though it has many omissions and contains much that is apocryphal, or at least wildly inaccurate, it scores over the older, more pedestrian work in two important respects.

First, it is slightly cheaper; and secondly it has the words "anyone can edit" inscribed in l

Re:2X

[Brian Gordon (987471)]

Ahaha, 2.9GB? That's the text alone. Images will net you more than 200GB [wikimedia.org] more. And yes, you do need a LAMP/WAMP and working mediawiki, but it wouldn't take 'days' it would take a few hours max. Also is this guy aware that wikipedia is available on DVD [wikipedia.org] already?

Re:2X

[TubeSteak (669689)]

Also is this guy aware that wikipedia is available on DVD already?

Are you aware that the link you pointed to (1) is not the same thing as the link (2) the author pointed to?
(1) http://schools-wikipedia.org/ [schools-wikipedia.org]
(2) http://download.wikimedia.org/enwiki/latest/ [wikimedia.org]

1 is 4625 articles hand picked for school age children, hence the website name
2 is a straight dump of wikipedia

Just imagine my surprise when the schools-wikipedia website didn't have the wiki article on Goatse!

Re:The Point?

[Mr. Roadkill (731328)]

I know that not everyone has a permanent connection to the net everywhere they go, but what is the point of storing a local copy of Wikipedia?

Ummm... I think the whole point is, as you've pointed out, that not everyone has a permanent connection to the net everywhere they go. Or maybe they don't have access to everything they'd like even if they *do* have net access everywhere, or want to pay extravagant data rates while out and about.

Joe has all-you-can-eat broadband at home, or an understanding employer with a fat pipe, and spends two hours each day on the train. Two and a half gig per month (and lets face it, you probably don't want to update it more frequently that that) and he's got probably half his reading material sorted out.

Wang lives in Buttfuckistan, a fictional country with totalitarian leanings with too many real-world counterparts. The Great Firewall of Buttfuckistan (i.e. squidguard, under the control of Buttfuckistan Telecom, and settings in the routers to drop non-port-80 traffic half the time) makes it impossible to reliably access Wikipedia from inside their borders, which is a great shame because the entry on Buttfuckistan is particularly unflattering. Once a month, Joe sticks a DVD with five minutes from an old re-run of Friends and an encrypted dump of Wikipedia in an airmail envelope and sends it to Wang.

Mary is still at secondary school, and her particular school has wifi access for students who are encouraged to purchase their own laptops, but since the local pastor discovered http://en.wikipedia.org/wiki/Image:Dream_of_the_fi shermans_wife_hokusai.jpg [wikipedia.org] they've been forced to add wikipedia to the school's blocklist. Which is a pity, because it's a great first-approximation source for material or research directions, but there you go. Mary can make a local copy through her home broadband connection, and can access it locally on her laptop wherever she goes - even at school, or church. Bill, Jillian and Mungo (the pastor's son) find out about this, and now all four of them take it in turns to make the copy each month, sharing the bandwidth costs. Their friends Harry and Sally, who don't have broadband but are great friends of the other four, also get copies... and there are plans to distribute the copies further, as a kind of teenage grass-roots knowledge-sharing and social-justice effort.

Still can't see the point?

Re:

[pla (258480)]

Do you mean searching takes days, or loading? Searching should be quick if you index the words. If you are duplicating a bunch of local clones of wiki, then simply copy down the raw MySql table data files rather than reload from delimited files etc. (One needs to make sure their version of MySql is compatible with the table file format.)

I suspect the former, plus creating the index, plus the not inconsiderable overhead of running an SQL server.

DBs have their place. For a "real" Wiki, or more generally