Jump to content

Master Database And Mirror Sites


expat-global

Recommended Posts

There is a lot of room for improvement in the web/email based information sites as well as telephone based, and the following is an "open source" project suggestion.

Instead of everyone doing a lot of web searches for information on particular people and posting all over causing repetition, the information sought and available (two databases) should be systematically organized into two master databases and all websites should point to these two to every visitor.

Instead of everyone going to a few websites which get overloaded and often crash, there should be a standardized database which can be distributed to various websites to reduce the load.

Then a list of participating websites could be made available.

Having worked for both a national government and also having been involved in UN and NGO international relief efforts, I can say that many government and NGO people do their best, but the very nature of these bureaucracies and approval systems can make them very slow and often the conceptual designs not completely adequate.

In a tragedy like this, the government people should share their database information.

This is not a major effort as regards custom programming, web hosting, and call centers.

The most difficult first step is getting people to move away from browsing and quick postings, and towards an organized, systematic, and simpler process.

The information from various sources needs to be put together. A standard database with many fields is created, and NEW information from various sources (hospitals, embassies, web visitors) is imported into this database. Of course, the data won't come in the same format, but it is easy to reformat it for import, or even to just have someone retype it.

After each round of updates, the master databases can be distributed around to the mirror sites.

As I see it, there can be one database, but there might be two databases: people accounted for (safe, injured, and dead), and missing people.

There might even be a third database for expats who were not known to be in the affected areas but who just want to say "I am OK". For example, even though I work and live in Bangkok, I received emails from people asking if I and my family are all OK. I also run a website here, and I have received requests to contact expats who are supposed to be in Isaan and other places but their relatives want to make sure they are OK and didn't go to the Andaman Sea -- because they haven't contacted home. This third database could be separate in order to reduce the work in the first two. NOTE: CONTACT YOUR FAMILY AND FRIENDS.

The databases should be simple PHP format and easy to install.

They should export to simple text or very simple HTML files for sites that just want to list the file for a Ctrl-F text search, rather than install the database (which isn't simple for many webmasters, and would lose them). This text version can be distributed, too.

Instead of linking to someone else's website, you can put it on your own and help out with the load. Importantly, on the link, you should state the date and time of the latest update.

As phone call inquiries are in much greater numbers than website inquiries, and most people affected are not as computer literate as us, there will need to be call centers where people can phone and ask a computer operator. The computer operator can just get on the web and go to one of these sites. Or their own site. Or their own hard disk. Or their own printout in alphabetical order.

The fields of the database (and of course you can add to this):

Source(s) of data (particular hospital, embassy, web visitors, etc.)

Source's record number

Assigned record number (by our own database)

Status (safe, injured, dead, missing)

Nationality

Surname

First name

Gender

Birth date in ##-Month-####

Age (don't make people do math on the birth day)

Is there specific information this person was expected to be beside the Andaman Sea? (Yes/No)

Believed regional location at time of tsunami (Phuket, Krabi, etc., or "unknown")

Known specific location (hotel, beach, etc.)

Mobile phone number

POINT OF CONTACT, name

POINT OF CONTACT, email address

POINT OF CONTACT, phone number

End of record, but you can suggest additional fields.

It is important to know that there are many people who don't use the internet at all, and countless more who don't know how to use it well. In many cases, the affected family is depending on a relative, friend, or other associate (such as a call center) to do the searches.

In fact, calling the affected family to ASK if there is an update may not be the best thing to do. These people are often jumping up in hope every time the phone rings. It is often better to have a designated POINT OF CONTACT, and everyone who is worried about their family member should call that point of contact.

We should not get complicated at first, but should get it out.

After this is up and running, then we can look at specific things like generating custom reports, e.g., by nationality.

Again, this is not a complicated project. Don't make it more complicated than it needs to be. The main challenge will be getting people to switch from browsing & semi-random postings to a disciplined project.

I can do some of these things, but some I cannot do. I run my own two servers, which are already a bit overloaded with multiple clients' websites running on them; nonetheless, I can manage this if we can get a few ISPs to donate servers temporarily -- ISPs in both Thailand and overseas. I cannot do PHP programming and I no longer have a PHP programmer in-house, but I have managed them before. (I can do only HTML, unix stuff, some Photoshop, but mostly project management.)

What do you think?

Who is willing to commit to helping?

Link to comment
Share on other sites

Will be difficult in getting the info into the database not making it. I agree that the current ones are lacking a lot.

Right, there would need to be one master database and database administrator(s).

The best I can find is www.missingpersons.or.th and they would be the first people to contact.

Notably, any links to their site for foreigners should go to

http://www.missingpersons.or.th/index.en.html

as otherwise it is incomprehensible and will lose many people.

The database on www.missingpersons.or.th is very well designed. One nitty problem is that you must keep clicking on the "English Version" link in little letters on the somewhat top right. Nonetheless, this is overall a very good website, and seems to be one of the best places to start. It's very fast (!!) and seems to handle the traffic much better than a lot of other sites.

www.missingpersons.or.th is operated by NECTEC (but clicking on their link at the bottom of the page fails, though you can just type www.nectec.or.th/english to get to them, whereby you are given their info@ and webmaster@ nectec contact addresses).

Internet Thailand is an ISP which is volunteering and has a good English language information page, and maybe they could be a mirror since they are a large ISP built for this kind of traffic. I tried to send them a message, but there is no contact link on their English language page, and going to their home page and finding the "Contact Us" link at the bottom, when you try to submit your contact form you get a 404 page does not exist. (They overhauled their website over a month ago, and it still doesn't work well.)

For some other websites, there is no contact email address, and no contact form.

At least NECTEC is in my province (Pathum Thani), so I can resort to "sneaker net" -- driving there and walking in the door. If it's open today. My office is closed, and I finally have some time for this.

My inclination is to work with NECTEC as primary project coordinator. Historically, they have been a leader in Thailand, and I know they have the bandwidth and computer resources to do this, unlike many volunteering websites.

To make this work without crashing our servers, I think we need to distribute databases or HTML files to many cooperating websites at once, but new entries be made directly to www.missingpersons.or.th (and have them improve their home page for user friendliness).

Link to comment
Share on other sites

Moving this topic to "general Topics".

/Admin

Why did you move this out of the Tsunami section and into the General section? We are much more likely to get practical help from visitors of the Tsunami section who are the specific subsection of your visitors interested in this particular topic, but not much from the General ThaiVisa section.

(For example, on my websites offering help on the Tsunami issue, I like directly to the ThaiVisa forum on Tsunami, not the general ThaiVisa forums.)

Link to comment
Share on other sites

My inclination is to work with NECTEC as primary project coordinator. Historically, they have been a leader in Thailand, and I know they have the bandwidth and computer resources to do this, unlike many volunteering websites.

I agree. They also have the political clout and mandate from the government to make it happen, and have interests in things like flood control and other risk mitigation stuff in rural areas. Recently part of NECTEC was spun off into the Hydro Agro Informatics Institute. I'm not sure which functions went where (HAII is now a separate office in town), possibly they would now be the relevant body.

I'd really be amazed if they aren't already working to consolidate data though, they're a pretty sharp bunch. If there are problems with the website, might be worth just emailing them in with a 'please fix'. Or encouraging other sites to offer their data to them.

Edited by Crushdepth
Link to comment
Share on other sites

hi,

I have a good server (about 5 gb) full of shit about my work (mostly past projects) I can free 4 Gb and use it as mirror.

bout DB, I am more limitated, I can only offer a Mysql.

I am late to answer this post, I worked offline all the day, but I would like to do something better than to complain about this or that (my usual way).

also, if any of my limitate skills can help, I will be more than glad to do it.

E.B.

Member of IETF (www.ietf.org)

Link to comment
Share on other sites

I'm a little late in responding here but I already responded by email that I'm willing to help wherever I can.

Unfortunately, I am now overseas for the next three days so will not be that accessible.

However, I do know who is responsible for missingpersons.go.th - and I think his intention is to do very much the same thing. I was in contact with him yesterday.

My suggestion would be for expat-global to get everything ready and post here - and then I will go to the administrator of missingpersons.or.th and see if he can make it work (and based on my knowledge of this person, if there is anyone who can make things work, it's him). Plus - there is like a 99% likelihood that he can rope Inet-TH in as well.

Edited by onethailand
Link to comment
Share on other sites

Unfortunately, I am now overseas for the next three days so will not be that accessible.

However, I do know who is responsible for missingpersons.go.th - and I think his intention is to do very much the same thing.  I was in contact with him yesterday.

My suggestion would be for expat-global to get everything ready and post here - and then I will go to the administrator of missingpersons.or.th and see if he can make it work (and based on my knowledge of this person, if there is anyone who can make things work, it's him).  Plus - there is like a 99% likelihood that he can rope Inet-TH in as well.

I've got only a few days before I must return to work and direct projects again, so if this is going to roll, the time to do it is in the next 3 days. (Also, 3 days is a long time to a lot of people around the world and the evolving process at this time of crisis.)

I know Dr. Thaweesak Koanantakool way back to 1994 (who became Director of NECTEC about 5 years ago), and have various other relevant contacts, as does my Thai wife from her years as a journalist. We are somewhat outdated, however, and you know how old email addresses are overrun with spam and phone numbers obsolete. We have analyzed this and come up with various plans, and will give this a go Saturday. Any way anyone could save us some time with some phone numbers or direct email addresses would be appreciated. Don't post them here! I can be reached at mark2 at permanent c0m (but ignore all email addresses at my website, too much junk mail, hence the 2 in username ... and phone numbers obsolete).

Here, what I think we need most is VOLUNTEERS of all nationalities, especially those residing in their home country, who are willing to TRANSLATE an essentially 1-page website into their native language, contact their embassies and national governments to present this system in order to assist them, help push to disseminate the information in their own countries by various internet and non-internet channels (promotion and PR), maybe take some phone calls, and try to get the government and NGO people who normally take the calls to use the system.

Please understand that most governments are rather bureaucratic and slow, with top decisionmakers and directors being relative computer neophytes and delegating technical matters to established career government techies who are sometimes not the best at technical implementation as regards user friendliness, practicality, or promoting it in their communities -- especially in a short time. I'm most happy when I'm wrong about this, but historically it's usually dismaying. I've heard and read a lot of complaints, including in CNN and other major media reports.

It's usually best to push along all 3 lines -- government, NGO, and purely private sector.

Right now, it seems that NECTEC is again a leader of the pack in Thailand, with its MissingPersons.or.th

The hospitals on the west coast and a lot of other people have done very good work getting lists of the injured, dead, and found up on the internet.

The big challenge seems to be the missing persons, and identifying the bodies.

It seems that the NECTEC system might be able to handle the bandwidth and the I/O crunching, but I don't really have enough of a basis to say they can. There may be significant surges if/when it hits CNN.com and other news sites. It's best if PR is done country by country. Look what happened to the Red Cross website when it hit CNN. Practically dead for a long time.

I've had super fast websites with a lot of traffic suddenly slow to a crawl and die completely after a national or international news report.

The best thing is to have local websites in each respective country and each respective language, and do just local PR within one's own country with one's own national news sources. Little surges, compared to international news coverage.

Browsable reports in HTML format exported from the NECTEC database may be sufficient, e.g., of all Norwegians on a Norway based website. Just static files updated periodically. Besides missing persons, the same applies to injured, dead, and known OK. These should be checked first. Simple HTML files are the simplest and most likely to succeed way, plus they bog down sponsoring websites less than PHP/SQL.

These static files can be distributed to multiple websites to be promoted by the various webmasters. Must be date and time stamped.

However, inputs of new missing persons is a different matter, as Darknight noted, and I currently think they should be done at missingpersons.or.th only, unless we get ONE established and reliable central database in each country, e.g., in conjunction with a government or established NGO.

The missingpersons.or.th database is missing some important fields, which could be made up for by national databases on-line.

I wish we could always depend on the governments and their established contractors, but my own personal professional experience working in international communications network circles in Washington, D.C., and elsewhere (USAID, UN, NGOs) is that they are too slow and often poorly conceived and implemented, even though it doesn't take a whole lot to do. I've created networks from the ground up in a very short time, brought in during a crisis after the established ones costing orders of magnitude more and taking years to develop couldn't do the job. Sorry, but this is fact, and those who don't learn from history often repeat it. I know there are people reading this forum who can contribute significantly if you just commit to trying, and it's better than spending one's time watching modern tribal warfare (e.g., football) on New Years Day. ;-)

Any critiques and other inputs of this are most welcome. This is the greatest thing about the internet -- it makes people accountable, including government. To have freedom of the press, you no longer need to own one. And YOU CAN MAKE A DIFFERENCE with work yourself -- it is easier to criticize others than create oneself, so I think it's best to be constructive. All you need is a computer, an internet connection, some smarts, go-getter initiative, and good intentions. You don't need to be a techie, if you have a good techie friend or colleague. PR can be a meaningful community experience.

Happy New Orbit.

Hopefully we will have a tsunami warning system by the following New Year, before we start hanging out at the edge of the hydrosphere again.

Thailand is a naturally beautiful place with some of the most hospitable indigenous people in the world, as witnessed by the large number of foreigners in Thailand in the worldwide news reports who chose Thailand as their holiday destination. Let's help each other like the hospitable Thais, strengthen the network bonds worldwide, and get to know each other better.

All the best,

Mark

Link to comment
Share on other sites

OK... LOL... you take care of the arrangements and I will lend a hand as soon as I can :o

K. Thaweesak's email address should be the same one he has always had - which I am not posting here for obvious reasons.

As for usability/feasability/load, let me think about that a bit. Not so sure putting everything on a single page is the answer but obviously one page for each letter of the alphabet is way overkill too.

it's better than spending one's time watching modern tribal warfare (e.g., football) on New Years Day.

Speak for yourself... LOL... the Trojans are bidding for a second straight national championship - they are going to maul Oklahama.

Sadly, however, I don't think they will be showing the bowls where I currently am (Taiwan) so I'll just focus on work when I'm in.

Edited by onethailand
Link to comment
Share on other sites

This forum subsection, "Tsunami politics, economics and comments" is a restricted "Protected Forum" so that guest visitors cannot browse it unlike practically all the other ThaiVisa forums. (I.e., those of you here reading it are Registered, Logged In users.)

Is this necessary?

Link to comment
Share on other sites

it's better than spending one's time watching modern tribal warfare (e.g., football) on New Years Day.

Speak for yourself... LOL...

We all need a break, and I suppose sports is instinctually one of the most gripping distractions from all this misery coming in on the news, web, and emails. If my Alma Mater were in a sports game, I might feel differently. Perhaps another "gripping" break is a few beers on New Years Day. Then maybe Animal Planet Live in Sukhumvit bars, watching the migrant mating games, that overriding instinct to deposit one's DNA into the pool, reinforced as the #1 instinct over countless generations of those most successful. Then there are the ostentatious status-conscious to entertain us when people-watching. We are a funny species, so much like the rest in these ways.

The Andaman Sea is one of the most beautiful ecosystems and terrains on this living and evolving planet, and one of my most recommended places in the world for vacationers.

Then, a tsunami hits and we are shocked into peoples basic needs, yet dependent on government bureaucrats.

After missing people, it's rebuilding time.

Enjoy the break while it lasts ... as your www.OneThailand.com website is a great piece of work, and I'm sure you need a break, as we all do!

Edited by expat-global
Link to comment
Share on other sites

This forum subsection, "Tsunami politics, economics and comments" is a restricted "Protected Forum" so that guest visitors cannot browse it unlike practically all the other ThaiVisa forums.  (I.e., those of you here reading it are Registered, Logged In users.)

Is this necessary?

I'll have a check with george. I haven't noticed this thus far. i thought is was open but maybe he has reasons not to open it ??

I'll get back to yah

Link to comment
Share on other sites

Rather than setting up an additional database, I was thinking along the lines of writing a program to spider the main sites, and pull their info into a database that could cross-reference the text, to try to form a 'super-site' of links. Any views as to whether it could provide anything useful?

Link to comment
Share on other sites

Rather than setting up an additional database, I was thinking along the lines of writing a program to spider the main sites, and pull their info into a database that could cross-reference the text, to try to form a 'super-site' of links. Any views as to whether it could provide anything useful?

Ultimately, that would be a good solution, if not for one big problem - every single database is different in format and it would probably take ages to get it right.

I think the brute force technique of recompiling the data might be easier...

Link to comment
Share on other sites

I've discovered something else which needs immediate attention, maybe more so than the master lists.

Krabi Hospital's photo gallery of unidentified victims is huge - and either very poorly organized, or they have a severe lack of bandwidth, or both. There are pictures of around 400 bodies, some of a reasonable size, some quite large (100K+) and they need to be forcibly dragged from the server, thumbnailed, and then a list of thumbnails put up which link to the correct pictures.

At present I just can't get any of these ###### things to load, let alone run software to drag all the pictures out - and besides I don't have enough storage or wide enough bandwidth to host a mirror of these pictures - but these bodies need to be identified and may at least put some people's minds at ease that the body of their loved one has been discovered.

Following are the raw URLs to the directories where the pictures are located. Would one of you volunteer to at least drag the pictures off their site?

http://www.kbo.moph.go.th/dead

Make sure you follow all the directories as well (1-5 at last look). I might be able to help set up the gallery/thumbnailing scripts but someone else has to host it, preferably with lots of bandwidth.

Link to comment
Share on other sites

This forum subsection, "Tsunami politics, economics and comments" is a restricted "Protected Forum" so that guest visitors cannot browse it unlike practically all the other ThaiVisa forums.  (I.e., those of you here reading it are Registered, Logged In users.)

Is this necessary?

I'll have a check with george. I haven't noticed this thus far. i thought is was open but maybe he has reasons not to open it ??

I'll get back to yah

Has been fixed

Link to comment
Share on other sites

Message from user in disaster forum.

Voluntary database, networking, and computer specialists from pantip.com will leave Bangkok for Khao Lak around Jan 4. Interested persons must confirm their availability by Jan 4 8.00am. Send your full name and contact number to [email protected] or call 01-4938900. http://www.pantip.com/cafe/lumpini/topic/L...7/L3208287.html (Thai language)

Urgently needed on spot:

- Laptop and PC

- Switch and hub

- Access point

- External HDD 80GB with USB port

- Flash drive

- Wireless LAN

- Scanner

- Digital camera

- Farang-size clothes

- Gowns

- Food + bottled water

- Money

Link to comment
Share on other sites

I send you a private message, with theinformation concerning my server, up to you to use it or not.

I did not post that here, because I DO NOT want it could considered as advert (whatever I do no work for local usually).

One things, you will have up to 4 gigbyte, but the address will be something like :

my_server.com/your_materials_pictures

if you need a subdomain or a domain, I am sorry that I can not set up, but you can still link to there (it's also teh reason why I don't post my server link here).

Regards

Link to comment
Share on other sites

About parsers, it's possible to do it in 2 hours .... but it will be okie only for one site, I mean every site can be parsed only by the corresponding parser.

interrest, all the data are get in the same form.

problem, if there is many websites, it will take long ... maybe too long ... I thouhg about that 2 days ago, but I have to work, I have to eat I have to sleep ... This kind of stuff must be made by a team, but I am willing to coordonate :

I need a list of sites to parses

I can prepare the regex for every site

I can prepare the DB design

but after I will need help for testing, that's long, and I also need to have strong and precise feedback , not "sorry mate, that still don't work", but if we do that, it will be necessary to have technical feedback : that parse this text , this text, but not this one ....

I will be online certainly all the afternoon, let me know

Edited by sting01
Link to comment
Share on other sites

I can give you a FTP account at Thaivisa if you want. Please email me at admin[at]thaivisa.com to discuss it.

But we must be sure that we don't invent the wheel again. I am seeing a lot of databases online, but no coordination.

Link to comment
Share on other sites

What do you mean George?

If there is need to have someone that coordonate DB stuff work, I am ready to do it.

I stated many time I have place and I can free one or 2 DB for that.

I repeat for parsers ( am good in that) , each parser can be made in less than 2 hours.

I can handle the DB design.

I agree there is not the need for another duplicate DB, but certainly for one that centralise all.

I am ready to start over, but I don't want to be as Don Qixotte. I rather would like to be sure I will not me too duplicate the existing.

To start over , things that are needed are :

the list of existing websites (URL of the pages where the information is located).

Also people who willing to help.

regards

Link to comment
Share on other sites

But we must be sure that we don't invent the wheel again. I am seeing a lot of databases online, but no coordination.

Right, we need to coordinate in order to not waste precious time and effort, create confusion to some users, and possibly create a mess which possibly (from my experience) some webmasters start to abandon (esp. as work starts again next week and we get busy) -- someone must take responsibility for sorting out any problems in a timely way.

I think there are ways that private global citizens of the world like us can help quickly and with a better quality of service. We also need to push some of the officials and guide them. I know -- I've worked with these kinds of people and on disaster systems.

Yesterday (Jan 1) I got in contact with several people, including the aforementioned K. Thaweesak, plus the Project Manager of the official missing persons database, and also the chief of the entire project. This is a project still coming together.

Right now, I think that a website like OneThailand.com which focusses solely and specifically on the disaster, was one of the earliest and now one of the best known websites, and is a well done starting point on the private side, can be a good site for updates, and it already links to this forum.

About bandwidth, especially as regards images, please be aware that you may incur excess bandwidth charges (and I would bet that most bandwidth is by the curious rather than those trying to identify a loved one). Ask your ISP or rack farm owner for permission to go above your quota on a charity basis from them.

Notably, while most identifications may be by systematic DNA comparisons from family, there are still a very large number of identifiable bodies from photos.

Already, missingpersons.or.th slows to a crawl -- and times out -- when operating the database much of the time (thereby impeding official work), and what I am discussing with them is generating reports in HTML and/or plain text to circulate to all interested websites, listing the missing and their details, and a home page which refers the curious to these other websites in order to save bandwidth for officials and people reporting new missing persons or updating records.

After viewing the details of database records, people may be able to add significant information on particular missing people. It's important that the database is as complete as possible.

However, it's most important that the site not be overloaded, and that it continue to function for officials around the world and people making meaningful additions and changes.

Keep in mind that phone calls to embassies and other points of contact are still the #1 way that people inquire, as you may be surprised at how unskilled-at-internet many families are in Europe and elsewhere. Some webmasters currently in Europe might be able to help out with a regular phone line.

I will be back at work on Tuesday, but I run my own business as director so I'm my own boss and am not actually very busy with company work at this time. I've committed myself already to helping the abovementioned people, so I'll update y'all here as soon as relevant things are agreed and materialize. I'm pushing.

All the best,

Mark

Edited by expat-global
Link to comment
Share on other sites

OK.

Let me work on the thumbnails for the Krabi photos - they desperately need to be organized in a better manner than "a1, a2, a3" links.

I'll do a speed test after that and see if the backup server is fast enough to host the main images - in which case space on another server won't be required right away. If not, then I will definitely make use of one or more of the offers which have come in, thank you very much.

Re: the database, you might be able to write a parser which covers some databases - but having looked at most of these databases myself, the only ones which are similar actually contain the same data as well.

I will have to look at missingpersons.co.th to see what's up. expat, did K. Thaweesak agree with the plan?

The Phuket IT City database swelled massively to almost 30K names - but many are repetitive, and they are now timing out as well. I don't think they realize just how much resources a search will eat up.

I've been away at a family ceremony all day, and will be working as much as I can to complete the Krabi reorganization by the time I hit the hay tonight, as I have to get up early yet again tomorrow, pack my bags, go on one more family outing and then head for the airport - after which I will pick up where I left off.

And btw, expat, thank you for the kind words - but it came about because of the same desires and frustrations all of us are experiencing - lack of information in a somewhat organized manner. Just imagine how much worse it is for those people who are missing friends or family... that is what drives me and I'm sure the rest of you - so keep up the good work!

Link to comment
Share on other sites

Re: Krabi Hospital unidentified, they have put up a new server.  I am busy downloading the files now because they are still poorly organized.  I will thumbnail them - but I'm still looking for someone to host the collection.

You can Send them to my server in the states if you want (calpop.com).

Windows 2003 server, sql database running.

1000 gig bandwith monthly, about 800kb download speeds

I should have space enough i think 120 gb left.

Also dns running if you have a domainname.

Let me know

Currently only running my webshop asian-art-shop.com and some mail for me

Link to comment
Share on other sites

All offers appreciated - and I may still take you up on them but for now I have got them up and running at onethailand - thumbnails only, linked directly to the full pic on the Krabi server. If for some reason either server slows down, I will proceed with one or more of the other offers. Thanks again!

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.




×
×
  • Create New...