DB corruption... :(
Hey guys. I know this used to be a problem with alpha updates back in the 20a-30a range, but have there been any recent issues with SS dropping the scraping sessions from the database? Two times in a row this evening, I opened my workbench and my scrapes were gone, leaving only the scripts.
To be fair, I had Actions enabled at the time, and SS had frozen/died on the first DB drop I experienced.
Any insider info on possible things to avoid?
PS-- just happened again,
PS-- just happened again, being the 3rd consecutive time. Nothing fancy is going on my end. I disabled actions after the 2nd DB loss, before attempting to recreate my scrape. After calming down for a few hours, I rewrote my scrape and exported a dozen times during the process. I did a DB backup manually via the File menu. I saved and closed the program. After re-opening, the scrape is gone again. I restored the backup, but it's still gone.
I have a suspicion that it's not *really* gone, but that the workbench is unable to show me the file, and then just deletes it from the internal database. I notice that the file size of the in-use "ss.backup" file decreases in comparison with it's truly backed-up copy that I created. If I copy and paste my backup into the resource/db/ directory and then launch and close screen-scraper, the file loses a few hundred bytes. I'm completely unsure whether this matters or not. I've tried restoring the manual backup several times and it predictably never retains the scrape data between launches.
When I run my copy in server mode (workbench closed of course), the server acts like it's in limbo-- the registration button is visible at the top-right, the memory usage indicator never loads, and the interface ignores my clicks.
I'm sure the problem would go away if I were to remove screen-scraper and re-install, but I figured it was worth a post or two in the forum before I abandoned all hope :)
No one here has seen this
No one here has seen this happening lately, but if you can reproduce it, please let me know the steps and we'll investigate (we're on the cusp of releasing version 5, and would love to root out any bugs).
Did you make sure the DB process died as expected? Sometimes if it's still running it looks like a DB corruption.
Okay. So I've been playing
Okay. So I've been playing with this, and here's what I've got:
I'm getting the same behavior out my laptop, which did not have SS installed previous to now. If I transplant from my PC to my laptop the active DB files or my File->Backup DB files, my laptop refuses to keep scraping sessions, too.
I can send the DB files to you if you want. I really have no idea how they got like this. It began happening after I went through my workbench and deleted everything (there were no folders, so it was a one-by-one process), and then began a new folder and created my brand new scraping session, proxy, and scripts.
To clarify contextual info, my laptop will drop Sessions without any tweaking of configuration at all-- I've got all default ports in my properties file, etc. It's a vanilla installation, but with my other PCs flaky DB files.
Tim
Hi Tim, Jason passed this one
Hi Tim,
Jason passed this one along to me. My best guess as to what's happening is that your database comes from a different version of screen-scraper than the one you're running. Any time you transplant the database files around you need to ensure that the versions are the same. For example, you couldn't take a database from version 4.5.61a of screen-scraper and drop it into a 4.5 installation (and vice versa). This is even often true between the alpha versions of ss. Is there a chance that could be what's happening here? If so, I can walk you through how to ensure that your database is at the same version of whatever ss instance you decide to run. Also, in debugging this kind of thing it's good to keep an eye on screen-scraper's error.log file. If you see errors along the lines of records not saving because of missing columns and such, it's likely because your database is at a different version than your installation.
Todd
This is slightly confusing,
This is slightly confusing, however, because the problem began occurring after my update from 54a to 61a, when it was only one installation. After that moment, my DB would never let anything stick.
It is the case that the error.log shows lines about column counts not matching, although I counted and the column-to-value count matches. However, it says
Column not found: character_set, and the SQL query is trying to insertnullas its value.Here's the full stacktrace:
An error occurred while executing the SQL statement. The error was Column count does not match in statement [INSERT INTO scrapingsession ( name, cookie_handling, cookie_policy, http_client, max_http_requests, external_proxy_username, external_proxy_password, external_proxy_host, external_proxy_port, external_nt_proxy_username, external_nt_proxy_password, external_nt_proxy_host, external_nt_proxy_domain, use_strict_mode, folderid, notes,anonymize,terminate_proxies_on_completion,number_of_required_proxies, originator_edition, logging_level, character_set, date_updated )].
java.sql.SQLException: Column count does not match in statement [INSERT INTO scrapingsession ( name, cookie_handling, cookie_policy, http_client, max_http_requests, external_proxy_username, external_proxy_password, external_proxy_host, external_proxy_port, external_nt_proxy_username, external_nt_proxy_password, external_nt_proxy_host, external_nt_proxy_domain, use_strict_mode, folderid, notes,anonymize,terminate_proxies_on_completion,number_of_required_proxies, originator_edition, logging_level, character_set, date_updated )]
at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
at org.hsqldb.jdbc.jdbcStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.jdbcStatement.execute(Unknown Source)
at com.screenscraper.util.DataMain.executeInsert(DataMain.java:426)
at com.screenscraper.util.DataMain.executeInsert(DataMain.java:454)
at com.screenscraper.data.DScrapingSession.setScrapingSession(DScrapingSession.java:245)
at com.screenscraper.business.BScrapingSession.setScrapingSession(BScrapingSession.java:154)
at com.screenscraper.scraper.ScrapingSession.save(ScrapingSession.java:2415)
at com.screenscraper.scraper.ScrapingSessionManager.save(ScrapingSessionManager.java:1125)
at com.screenscraper.model.ModelMainFrame.handleAppEvent(ModelMainFrame.java:200)
at com.screenscraper.controller.ControllerMainFrame.handleAppEvent(ControllerMainFrame.java:627)
at com.screenscraper.view.ViewMainFrame.promptForSave(ViewMainFrame.java:1878)
at com.screenscraper.view.ViewMainFrame.access$000(ViewMainFrame.java:43)
at com.screenscraper.view.ViewMainFrame$1.windowClosing(ViewMainFrame.java:163)
at java.awt.AWTEventMulticaster.windowClosing(Unknown Source)
at java.awt.Window.processWindowEvent(Unknown Source)
at javax.swing.JFrame.processWindowEvent(Unknown Source)
at java.awt.Window.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
SQL was: INSERT INTO scrapingsession ( name, cookie_handling, cookie_policy, http_client, max_http_requests, external_proxy_username, external_proxy_password, external_proxy_host, external_proxy_port, external_nt_proxy_username, external_nt_proxy_password, external_nt_proxy_host, external_nt_proxy_domain, use_strict_mode, folderid, notes,anonymize,terminate_proxies_on_completion,number_of_required_proxies, originator_edition, logging_level, character_set, date_updated ) VALUES ( 'hey', 1, 0, 0, 1, '', '', '', '', '', '', '', '', 1, 0, '', 0, 0, 5, -1, 1, null, 1274810909272)
Column not found: character_set
It was only after this began that I introduced another installation of SS, but even then, I had the new installation updated to the 61a alpha so that the versions matched before testing the transplant.
Any other insight?
Hi, I have heard of a handful
Hi,
I have heard of a handful of occasions in the recent past where, when updating between alpha versions, the database didn't get updated properly. Unfortunately, we've never been able to isolate the cause, but we've always been able to resolve it by forcing it to update again. That is, in your screen-scraper.properties file set your version number to something like 4.5.60a, then allow screen-scraper update itself again to the latest alpha.
Would you mind giving that a shot and posting back the result?
Thanks,
Todd
I actually just updated to
I actually just updated to 63a (I was on 61a from a few weeks ago) and it fixed itself, including giving me back my mysteriously missing scraping sessions. I'm glad that it came back all by itself!
Looks like this particular case is solved. Many thanks. I'll keep my eyes peeled in case it happens again.
Tim
Ah, that's good news.
Ah, that's good news. They're are definitely occasional bumps in the road with the alpha versions, but eventually we iron out the wrinkles :)
Todd