Database Team

- Introduction

This section of the wiki is for communication about the technical development of our databases. We have four kinds of data we'll be working with:
  • Collection catalog items (Catalog Maintenance System click here.)
  • Content of our web sites (no management strategy as of 2015-07-15)
  • Constituencies, that is people and institutions with whom we are in contact. (CiviCRM)
  • Digital repository (image files, text files, executables)
The links will take you to discussions of these databases.

On this page we will post updates from our database team members. The Database Catalog Team has a separate development site they are using so comments here may be brief. To see beta versions of the data entry system they are creating along with Liza's comments, questions and critiques, click here.

Database Team should consult with Catalog/Metadata Team periodically to better understand the system they are developing from the user point of view.

- April 30, 2016 Update

Active Team members:
  • Anna Narbutovshi
  • Stan Crump (PCGuy)
  • Alan Wilson
  • Liza Loop
  • Laney McGlohon

Work Table
Please log your assignments in this table so we can stay coordinated.
Start Date
Ready for Test Date
Complete Date
Help Request

- July 15, 2015 Update

-- Database Team Members - At the moment we have:

Liza Loop
Alan Wilson
Don St. Dennis (inactive, summer 2015)
Bob Speray (maybe, Liza is checking back with him)
Stan Crump (busy until July 24th. Please don't contact him until then)

Each of us should have our specific tasks posted on our volunteer pages. I'd like to put a general description of what we are working on here, on this page, in the section immediately below. Below that, perhaps you could post your questions to each other. My intent is to have our questions public and in text so we all know what they are and can jump in with suggestions and/or answers even if now asked directly. We should also note if and when the question/problem gets solved and link to where to find something about the solution. If you prefer to use email for this try to copy the questions and links here.

-- Tasks for week of July 13-19

--- Liza

  • managing new volunteers
  • (but primarily writing a grant proposal for Cal Humanities Stories Project)

--- Alan

  • learning CiviCRM and Omeka
  • building drop down navigation menu for wiki
  • building Volunteer page template for wiki
  • getting up to speed on CiviCRM form design

--- Don

  • researching permissions and authorities to see how to pass them from WordPress/Civi to Catalog Maint. Sys.

--- Stan

  • working a gig until July 24
  • tuning up Cat. Maint. Sys.
  • answering questions on " " "

- September, 2014 Update

From Stan (Sept 18):
Are you needing to add items using the new code as you do your inventory? I started working on anther view this week, assuming that you need the code working. I can't tell you which view it is right now, but I can switch to the one that's most needed. Each "selection" on the new screen will be another table in the database that I'll need Andrey to write maintenance code for.

From Liza (Sept 10):
Stan, Andrey, Laxman
Please put a few notes about what you are working on this month here.

The team has started a private wiki to share technical information. If you are interested in working on our catalog and integrating our catalog (collections management) with our contacts database (CiviCRM) please contact Liza (liza at hcle dot org).

Team Meeting: Thursday, May 1, 2014, 2 pm PDT

trying out free meeting service.

Join the meeting:
On a computer, use any browser with Flash. Nothing to download.
On a phone or tablet, launch the app and enter meeting code: hclemuseum

Join the audio conference:
Dial a phone number and enter access code, or connect via internet.
By phone:
United States +1.860.970.0010
Access Code 444-827-683# (oops! not working)

try code 621 698 489#

Other international numbers available

By computer via internet:
Join the meeting, click the phone icon and select 'Call via internet'. A small download might be required

Report on Stan's work
Discussion of needed catalog advisers (Ron suggested a librarian)
Review of existing open source catalog software
Schedule next meeting - May 15, 2 pm

Supporting information

invited: Stan Crump, Bob Speray, Ron Wickersham, Dick Karpinski, Diana Morningstar (Liza has been in touch with the preceeding folks) Kate Greenway, Rick Collette, (not heard from in a while), Tom Trimbath, Liza Loop (HCLE staff).

Report on Stan's work

Discussion of needed catalog advisers (Ron suggested a librarian)

Review of existing open source catalog software
Our Archive Consultant, Elizabeth Knight, recommended we check out these packages and not reinvent the wheel. Even if we don't use any of them we can probably learn a lot from their structure and metadata.

Schedule next meeting
Do we need to meet every week or is less often better?

The Team (invited to initial online meeting 24 April 2014, 2 pm):

  1. Liza Loop - Vision Keeper - working hard, extending my all too scanty knowledge - coming
  2. Jon Greene - no longer available
  3. Bill King - got a gig, bye bye.
  4. Tom Trimbath (coordinating people) - coming
  5. Bob Speray - Omeka consultant currently volunteering for HCLE and working on a book for Omeka users - coming
  6. Rick Collette - Volunteer on hold (who came through Craig's List) - no response
  7. Stan Crump (pcguy) - coming
  8. Kate Greenway - no response
  9. Chris Hoke - no response
  10. Dick Karpinski (nitpicker77) - busy, deceased Jan 11, 2015, goodbye and thanks for all the fish.
  11. Ron Wickersham - coming

20140424 Meeting Notes (most recent at the top):

Liza's report of meeting:

Tom, Stan and Bob, great meeting. Sorry I didn’t manage the technical angle very well but we got some major objectives accomplished:
1. We know each other better and understand some of what each can contribute
2. We discussed pros and cons of several database structures. In particular we explored the idea of HCLE being an aggregator or indexer of files served by other web sites as compared with copying all the files we want to exhibit onto our own server.
3. We clarified what we are going to do next:
Bob - put together a funding proposal for creating an image gallery in Omeka
Stan - continue to implement the MySQL system for manual data entry
Liza - work on clarifying uses and sources
Tom - has enough on his plate.
Ron - (from phone conversation after the meeting) look for holes in Liza’s vision and offer solutions. So far:
  • Bring in a librarian to ensure that our fields conform to the Dublin Core
  • Find an information shepard/moderator to keep the wiki organized
  • Back up our HostGator digital repository once a week to Ron's computer to protect files against accidental loss

24 April 2014, 2 pm

1. Self Introductions with emphasis on database skills
2. Each person's understanding of the database goals (briefly)
  • Audiences
  • Structure
  • Search screens
3. Discussion and agreement on a single set of goals
4. Suggestions for list of tasks
5. Assignment of tasks
6. Date and time for next meeting
Present at the meeting (online)

You have all met me, at least on line and we have talked about goals for HCLE as a whole and the database/catalog effort which is the center of the HCLE infrastructure. I am a dilettante in databases with just enough knowledge to get me into trouble and no ability to get out of it. My wish is to turn this problem over to this team and have you present me with a workable solution in, optimally, a few weeks.

All items in the database should have PURLs (permanent URLs) or other shared locators as well as reference to backup files kept offline.

I envision HCLE's audience to have 3 segments:
1) Staff and volunteers who build the database, enter much of the data, correct and expand the metadata and search the database for items to include in exhibits they build.
2) External researchers/scholars (young and old) who search the database for items of interest. These folks would have some idea what they are looking for and can use a keyword or element search to select a subset of catalog items that match their criteria.
3) Casual museum visitors who don't know what they are looking for. They will need a "flat gallery" interface, similar to that provided by Omeka and other display tools, so that they can browse through our images and dig for more information about the items they find interesting.

"Exhibits" are
1) collections of images presented in a coherent order with stories and commentary wrapped around them
2) collections of executable software with instructions/suggestions for use, commentary about historical usage and feedback mechanisms for research on user experience
3) later (much later) we may be able to simulate a physical museum in a virtual world setting


From 20140421 email: In the meeting, I'd like to discuss
  • how the data in the database will be used. If there are several collections of data that would need to be access together, but not with all the other fields, then there may be some benefits to having smaller tables. I believe this would make database access more efficient, unless we are always needing to access data from multiple tables, then the single table would be the best approach. MySql is very efficient, so I don't really see any problems with one big table. Creating the appropriate indexes is the key to efficient queries.
  • A short discussion about how the search screen might look would also be helpful in designing the database



Here's the screen shot Stan sent me of what he did a short while ago. I understand this view is generated from a MySQL database. Let's get a more technical description of the fields.

Clarifying email exchange between Stan and Liza from 20140417 (April 17):

Stan:I am very interested in working on the catalog. I've spent just under 10 hours so far and created the database and input screens for the identification table. I'm attaching a screen shot for you to see. I still have a little more work to do before its fully functional. It will take me some time to put this out where you can see it and try it out. I will try to do that as soon after all the minimal functionality is in place.

LL: I think we have a confusion of terms. In my (and MS Access) lingo, a screen is a view or a report generated from one or more tables. All the "screens" you see on the wiki are from a single large table with approximately 96 fields (or elements or columns). It looks from your text like you are interpreting each screen to be a separate table. That isn't necessary. Note that the same information appears on several "screens". I did this so that the user would not get confused about which record was active. Every record can be viewed via 5 or 6 screens with different subsets of the data displayed.

Stan:One question I have concerns searching for things in the catalog. I believe we'll need a keyword field in some of the tables that can be used for quick searching. I think I saw a field in the "data" table for that purpose. Do you think we need to have keywords for any of the other tables?

LL: Yes, we do need keywords. Look at the "Subject-Topic" screen. There are two fields for keywords, one labeled "Subject or Topic", the other labeled "Subjects and Keywords". My thought was that the first field would be a single value from a lookup table that corresponded to familiar academic-style subjects. The "Subjects and Keywords" field would be like a tag cloud that could accommodate several values. Since there is only one table that contains a single record for each item in the museum, these keywords could be used across the whole catalog. I anticipate creating several different kinds of search applications to query the database. At first all we need is for a user to be able to see a list of all the fields and fill in a few values -- e.g. Type: software, Subject or Topic: Math, Audience: 4th grade - US, Medium: floppy disk. This query would return all the items in the catalog that meet this criteria. The initial report would include: Title, Author, Thumbnail image, URL to the online item and physical location of the actual artifact. This is our basic scholars and staff interface. Later we will construct much more appealing museum display interfaces for everyday visitors. Every item should also have a unique ID number that can serve as a key for linking tables. Let me know if this makes sense to you. I'm not a database expert, I just know how I envision using the material once the database works!

Stan: As for the work, assuming that the perfect job-for-pay doesn't come along and consume all my time, I think I'll be able too build all the maintenance screens without additional resources, if that works for you. I know you must be wondering what my capabilities are and you need to see more before you probably feel comfortable with that, but I think the database and web maintenance could come fairly quickly.
That would be great. We are working on a proposal to the National Endowment for the Humanities so there is some possibility - no promises, that this could become a job in the future. To that end, please either post your resume on the wiki or send it to Tom so we'll have it if we need it.

Stan: I got a message from Tom Trimbath offering to contribute. I saw where his title had the words "Project Manager" and that put a big smile on my face. As we discussed, I believe, I've mostly worked alone over the last few years and probably am a little weak about leaving a good trail and communicating with others. I believe that is a strong skill of a Project Manager. Please guide me, as I need to reply to him soon. It looks like you and he are "the staff" for HCLE. What role would you think he would like to have on the catalog and if you have any suggestions about what expectations he might have, I would like to know and respect them.

LL: Tom's...role is to facilitate communication among the database folks as needed. He is a true gem -- an aerospace engineer who can write, create pert charts and loves social media -- a rare combination. Although not a database professional Tom is both bright and appreciative of other people's talents -- and he catches on quickly. He is looking forward to keeping the database team of stray cats coordinated. And yes, he is currently our only paid consultant. You can say anything you want to him. I do.

Stan: Let me know what questions you have and what your thoughts are. My work motto is "chose work you enjoy and always make sure you and the boss are on the same page". Tell me if I need to turn a page or two. ;)

LL: Sometimes it takes a few turns to verify that the pages are the same. We'll get there. There are about 6 other folks who have expressed interest in the database team. We'll have a meeting as soon as 4 folks respond that they can get online at the same time. (Thurs, April 24, '14, 2 pm PDT). You, Tom and I make 3. Keep your eye on Doodle to see who else can synchronize with us. I'm really pleased that we are moving ahead.