Master Database And Mirror Sites

[expat-global]

The Phuket IT City database swelled massively to almost 30K names - but many are repetitive, and they are now timing out as well.  I don't think they realize just how much resources a search will eat up.

<{POST_SNAPBACK}>

A lot of responsibility has shifted to ThaiTsunami.com by the Ministry of ICT, and thank goodness it's a dot com.

This website is well designed, but the data is still being prepared.

Note how many in the "Missing" have a full name but still have N/A for their country. Even age is zero, and approximate would help. Just getting inputs from the public via internet about THAT could help, where in fact the information is truly not known already...

... because, as you can see, they haven't got anywhere near all the missing persons entered yet (as of the time I'm writing this), nor the data standardized between the different sources (e.g., surname but not first name for some records), nor a way to click on any person's name to get any more detail, even a yes/no field on whether an adequate contact person has been established about a particular missing person.

That will come real soon now, but this is a new direction in which the missing persons project seems to be heading.

Note that inputs are by email to an operator's email address, or a phone call to a prominently displayed number. Not database driven, nor even a contact form. There is limited database driven material. However, the links are database queries, not HTML or text files.

About the images project being discussed here:

There still could be a master photos link for identifying people for where there is a photo but still no identification of them, and as you noted, that requires thumbnails with links to large photos.

Yes, bandwidth is just one of the issues, as database queries and i/o crunching take up CPU and other system resources which turn out to be the bottleneck, not bandwidth, in many cases. I suggest HTML pages to go with the thumbnails and photos, though of course that set of pages can be exported from one database into a tar or zip file before distribution. Moving MySQL databases between systems also requires a bit more skill.

Notably, I am not an expert at this, and you probably know better, so take my advice in the previous paragraph with a big grain of salt.

The images system and anything else can always be distributed and tried on-line on one's own server, to everyone who requests. It seems you all have good systems. However, if it bogs down one system, they can always replace the page with an auto-forward link to another system which can handle the load.

There is a lot being discussed in official circles, but decisions still pending, and a lot of people involved...

Phuket IT City's website is really commendable.

All the best,

Mark

Edited January 2, 2005 by expat-global

[expat-global]

[A lot of responsibility has shifted to ThaiTsunami.com by the Ministry of ICT, and thank goodness it's a dot com.

Correction: this may not be accurate at all.

The fact that MissingPersons.or.th works better carries more weight in my opinion.

Good to send suggestions to them all and enter dialogue, but it will probably come down to which one actually works technically the best and the first.

The same databases might appear on multiple sites.

ThaiTsunami has the better interface.

MissingPersons works better.

On ThaiTsunami.com, their records on-line are so far behind that we won't know where they stand until they get the updated database on-line and put to the test.

ThaiTsunami.com is asking for inputs by email, whereas MissingPersons.or.th has an on-line form to fill in which prompts you for your inputs.

Overall, it's good that there are multiple sites and databases out there under development, because we don't want to depend too much on either one authority or on one particular technical system.

Let's see which one works well enough first.

Right now, MissingPersons seems to be in the lead on-line because it works better technically. Easier and quicker to fix up the interface than to fix up the programming.

[Darknight]

missingpersons.or.th has been on the news in belgium as the official site for thailand as well as on th BBC

[expat-global]

missingpersons.or.th has been on the news in belgium as the official site for thailand as well as on th BBC

<{POST_SNAPBACK}>

Right. It also became official today.

Actually, ThaiTsunami is the "official site for Thailand" as regards general information, and MissingPersons.or.th is the official site specifically for the database of missing persons in Thailand.

Today, the MissingPersons.or.th site was updated at the bottom with "NECTEC is responsible for the registration system of missing persons."

ThaiTsunami.com was also updated just today. They have taken down their database today and added links to MissingPersons.or.th and Phuket IT City.

Unfortunately, ThaiTsunami added a flash splash opening screen, and have animated their logo at the top, which consumes bandwidth without giving useful and practical features.

On the plus side, the default of ThaiTsunami is in English.

Note that at the very top of the page is their announcement that they need telephone operators who can speak foreign languages.

Notably, MissingPersons.or.th was set up in only about 24 hours after a meeting with the Red Cross. It seems to work well. Proof that things can be done quickly.

MissingPersons.or.th have also implemented some of my suggestions on user interface. These people do listen, and are doing their best.

I would suggest that those working on the images system contact the relevant people with a conceptual design if not a working system to pass around. What works is used.

So many missing people, and for every one missing you can multiply many times for family, friends and associates overseas and in Thailand... The identification logistics by photo, description, and DNA is staggering.

I have some, too, as does my wife, plus some inquiries by email. We were staying on one of those beaches with our daughter on the previous month, and my wife had friends there who we visited... and all that's gone...

Mark

[onethailand]

I was actually talking about an urgent need to make the Krabi photos a bit more accessible, and now that's done. I hadn't thought about an images system but I suppose that could be done.

I am at the airport in Taipei on my way home, so perhaps I'll think about it on the flight.

[organic]

Rather than setting up an additional database, I was thinking along the lines of writing a program to spider the main sites, and pull their info into a database that could cross-reference the text, to try to form a 'super-site' of links. Any views as to whether it could provide anything useful?

<{POST_SNAPBACK}>

Ultimately, that would be a good solution, if not for one big problem - every single database is different in format and it would probably take ages to get it right.

I think the brute force technique of recompiling the data might be easier...

<{POST_SNAPBACK}>

I wasn't thinking of accessing their databases, which are probably private anyway, but rather spidering the HTML pages produced. There are, for example, a lot of blog sites set up for the disaster, which are very poorly indexed. I'm sure they contain loads of useful info, but it's hard to find anything on them. Google provides this function to a limited extent, but unless the site owners know what they're doing, they may not be getting spidered by it regularly, if at all.

[onethailand]

I wasn't thinking of accessing their databases, which are probably private anyway, but rather spidering the HTML pages produced.

Ultimately, that's the same thing, because the pages produced should reflect what's in the database.

Take my word for it - naturally I'd be happy if this could be done easily, and by all means if you think you can pull together all the scripts quickly enough, I'd love to see it - but having done exactly this in the past, I know how difficult and time-consuming this is - sure, once you get it right life is much easier but in this situation time is critical.

[organic]

Ultimately, that's the same thing, because the pages produced should reflect what's in the database.

Take my word for it - naturally I'd be happy if this could be done easily, and by all means if you think you can pull together all the scripts quickly enough, I'd love to see it - but having done exactly this in the past, I know how difficult and time-consuming this is - sure, once you get it right life is much easier but in this situation time is critical.

<{POST_SNAPBACK}>

Yeah, writing the spider's not so hard, but you need loads of back-end stuff too if it's not going to be a waste of space. Meantime I've started looking to see if there are local volunteers / companies want to help out as I'm currently restricted to dial-up access and a laptop that overheats rather too quickly.

[shaynevan]

There is a lot of room for improvement in the web/email based information sites as well as telephone based, and the following is an "open source" project suggestion.

Instead of everyone doing a lot of web searches for information on particular people and posting all over causing repetition, the information sought and available (two databases) should be systematically organized into two master databases and all websites should point to these two to every visitor.

Instead of everyone going to a few websites which get overloaded and often crash, there should be a standardized database which can be distributed to various websites to reduce the load.

Then a list of participating websites could be made available.

Having worked for both a national government and also having been involved in UN and NGO international relief efforts, I can say that many government and NGO people do their best, but the very nature of these bureaucracies and approval systems can make them very slow and often the conceptual designs not completely adequate.

In a tragedy like this, the government people should share their database information.

This is not a major effort as regards custom programming, web hosting, and call centers.

The most difficult first step is getting people to move away from browsing and quick postings, and towards an organized, systematic, and simpler process.

The information from various sources needs to be put together.  A standard database with many fields is created, and NEW information from various sources (hospitals, embassies, web visitors) is imported into this database.  Of course, the data won't come in the same format, but it is easy to reformat it for import, or even to just have someone retype it.

After each round of updates, the master databases can be distributed around to the mirror sites.

As I see it, there can be one database, but there might be two databases:  people accounted for (safe, injured, and dead), and missing people. 

There might even be a third database for expats who were not known to be in the affected areas but who just want to say "I am OK".  For example, even though I work and live in Bangkok, I received emails from people asking if I and my family are all OK.  I also run a website here, and I have received requests to contact expats who are supposed to be in Isaan and other places but their relatives want to make sure they are OK and didn't go to the Andaman Sea -- because they haven't contacted home.  This third database could be separate in order to reduce the work in the first two.  NOTE: CONTACT YOUR FAMILY AND FRIENDS.

The databases should be simple PHP format and easy to install.

They should export to simple text or very simple HTML files for sites that just want to list the file for a Ctrl-F text search, rather than install the database (which isn't simple for many webmasters, and would lose them).  This text version can be distributed, too.

Instead of linking to someone else's website, you can put it on your own and help out with the load.  Importantly, on the link, you should state the date and time of the latest update.

As phone call inquiries are in much greater numbers than website inquiries, and most people affected are not as computer literate as us, there will need to be call centers where people can phone and ask a computer operator.  The computer operator can just get on the web and go to one of these sites.  Or their own site.  Or their own hard disk.  Or their own printout in alphabetical order.

The fields of the database (and of course you can add to this):

Source(s) of data (particular hospital, embassy, web visitors, etc.)

Source's record number

Assigned record number (by our own database)

Status (safe, injured, dead, missing)

Nationality

Surname

First name

Gender

Birth date in ##-Month-####

Age (don't make people do math on the birth day)

Is there specific information this person was expected to be beside the Andaman Sea? (Yes/No)

Believed regional location at time of tsunami (Phuket, Krabi, etc., or "unknown")

Known specific location (hotel, beach, etc.)

Mobile phone number

POINT OF CONTACT, name

POINT OF CONTACT, email address

POINT OF CONTACT, phone number

End of record, but you can suggest additional fields.

It is important to know that there are many people who don't use the internet at all, and countless more who don't know how to use it well.  In many cases, the affected family is depending on a relative, friend, or other associate (such as a call center) to do the searches.

In fact, calling the affected family to ASK if there is an update may not be the best thing to do.  These people are often jumping up in hope every time the phone rings.  It is often better to have a designated POINT OF CONTACT, and everyone who is worried about their family member should call that point of contact.

We should not get complicated at first, but should get it out.

After this is up and running, then we can look at specific things like generating custom reports, e.g., by nationality.

Again, this is not a complicated project.  Don't make it more complicated than it needs to be.  The main challenge will be getting people to switch from browsing & semi-random postings to a disciplined project.

I can do some of these things, but some I cannot do.  I run my own two servers, which are already a bit overloaded with multiple clients' websites running on them; nonetheless, I can manage this if we can get a few ISPs to donate servers temporarily -- ISPs in both Thailand and overseas.  I cannot do PHP programming and I no longer have a PHP programmer in-house, but I have managed them before.  (I can do only HTML, unix stuff, some Photoshop, but mostly project management.)

What do you think?

Who is willing to commit to helping?

<{POST_SNAPBACK}>

Your thinking is correct. Fortunately what you are proposing is happening as we speak.

1. The Australian Government has promised $500 Million in aide. An important part of this is to set up in Phuket a DNA ID Lab. This lab will process data from all affected countries. It will also serve as a model for future events.

2. The company that provided the DNA analysis for 9/11 the Spanish Train disaster. Bali bombing -the world leader in Disaster Management has been retained by the Australian Government to lead the way in the Phuket lab.

3. The UN in conjunction with other relevant agencies will set up a Tsunami early warning agency in Thailand. Also a new "Early Warning" for private persons is available via Email.

4. Crisisnetwork.org is woking with various agencies to assist where they can.

a) creating a layer on top of all Thai hospital , govt and other databases to create a single source of information. creating a SMS early warning solution. c)creating other solutions for Disaster preparedness and recovery.

You may already know about the facial recognition technology that allows you to upload a photo of someone to imediatly compare against the govt database.

This is of course not so much, but many organisations are working together and Thailand seems to becoming their loction of choice for regional services. This can only say and reflect how highly Thailand is considered by the International Community

[organic]

I've been pressing on with my spidering / database idea, and it's starting to get places. Meantime I've put out an appeal for assistance and resources in the local paper today (you can't get anything done in the UK until well after New Year!). I've had a couple of positive initial responses. I can't promise they'll translate into anything definite, but please do let me know of anything else that needs mirroring.