It might be overshadowed by the ramp-up to LegalTech, but a big project of the Sedona working group on e-discovery will be kicking off later in February, the 2009 Text Retrieval Conference (TREC) Legal Track. For going on 18 years, TREC has been a workshop for encouraging research in information retrieval. The three-year-old legal track is organized through the National Institute of Standards and Technology and co-coordinated by Jason Baron, director of litigation at the National Archives.
Participating teams work with a test case to evaluate the most effective search protocols for finding relevant documents in e-discovery, i.e. what finds the responsive documents best? Concept search, expert human reviewers, Boolean keyword search using “x and y or z” or other methods? I spoke with Jason Baron briefly about this year’s TREC, which will switch test collections from the tobacco litigation “Master Settlement Agreement” repository to the Enron collection of email and attachments. He believes it will increase participation by reducing the need to search OCR’d documents.
Most of the participants in past years have not been vendors, but a few we know of are H5 and Clearwell Systems. It’s certainly a worthy goal to find the most effective methods and work towards improving standards in how we approach sifting through legal documents for relevancy. You’d think more vendors would be willing to put their tools to the test for the greater good, “walking the walk” if you will. Instead their absence is a reminder that e-discovery is still a wild west scenario with no “standards sheriff” in town.
How likely is it that vendors will put their products to the test in an attempt to back up claims and find better methods in review and analysis of Electronically Stored Information for trial? We hope that more of them will answer the call. It’s a good time to prove your chops if you’re in e-discovery – anyone going to LegalTech next week will be able to attest to the dizzying number of vendors making the market ripe for consolidation, which we have already partly seen through the high volume of acquisitions in the last few years. It remains to be seen how many of the smaller fish will still be left swimming at year end given the economic climate and fierce competition. Proving their mettle and working toward the common good wouldn’t hurt any.
I will be at LegalTech from February 2-4 talking to vendors about their products and their own methods for finding responsive documents. If you would like to get in touch or schedule a meeting, I can be reached here.
Subscribe via RSS
6 comments ↓
Nick – excellent piece and agree with the need for vendors to “walk the walk” – and would submit that the true “walk the walk” goes beyond the standards group – but into the actual “real world” environment support of standards AND the ethical commitment to honor those requests from clients to use “standards” (example XML Schema from EDRM) to transfer data to systems that may be competitive to their specific system. As a member of the EDRM and also a potential participant in this years TREC projects -we truly understand that their is a difference between “public walk” and “private walk” – and hope all vendors realize that the “private walk” is what will ultimately move the industry.
Hope to see you at LegalTech.
Rob Robinson (Orange Legal Technologies)
Dead on points Nick. While the technology used to identify information is important, how the tool is used is where the rubber meets the road. As filtering and review technology continues to mature and sophisticated feature sets become widley available, efficient methodologies in the use of that technology will become increasingly more important. Would love to visit with you more on the topic. See you at Legal Tech!
I don’t think TREC is the place for vendors to compare their tools, really. TREC itself is built on the notion that vendors who play can’t point to their performance, so even if they DID test their stuff, they couldn’t promote the results, which would muddy participants’ incentives substantially.
Also, we’re still sorting out WHICH KINDS OF QUERIES are best, as well as the critical ability to blend multiple queries into a cocktail. That’s where TREC could make a big difference. I hear you on wanting transparency in vendor comparison, but TREC’s not where it should happen.
Just to clarify, this is my opinion, not Nick’s, and I will be the one at LegalTech in case anyone is interested in a meeting. Thanks for reading and all of the feedback.
Katey
Nice article! And astute observations too. I think both the conferences are trying to achieve the same goal. But just trying to out do each other. The more the merrier I guess. Whit is right though. not getting to showcase your work at Sedona, IS a pain point. Maybe thats a cue for sedona guys!
e Discovery Tools Can Help Internal Investigations Internal investigations often require tools to help identify, collect, preserve and examine digital information.
e-Discovery tools were previously considered useful only in providing assistance for litigation, but that is not the only place they can be helpful.