From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 5256913877A for ; Mon, 4 Aug 2014 19:46:51 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 7D248E0937; Mon, 4 Aug 2014 19:46:45 +0000 (UTC) Received: from smtpq1.tb.mail.iss.as9143.net (smtpq1.tb.mail.iss.as9143.net [212.54.42.164]) by pigeon.gentoo.org (Postfix) with ESMTP id 4299AE08FA for ; Mon, 4 Aug 2014 19:46:43 +0000 (UTC) Received: from [212.54.42.134] (helo=smtp3.tb.mail.iss.as9143.net) by smtpq1.tb.mail.iss.as9143.net with esmtp (Exim 4.76) (envelope-from ) id 1XEODT-0006YH-4u for gentoo-user@lists.gentoo.org; Mon, 04 Aug 2014 21:46:43 +0200 Received: from 53579160.cm-6-8c.dynamic.ziggo.nl ([83.87.145.96] helo=data.antarean.org) by smtp3.tb.mail.iss.as9143.net with esmtp (Exim 4.76) (envelope-from ) id 1XEODS-0008Na-NR for gentoo-user@lists.gentoo.org; Mon, 04 Aug 2014 21:46:43 +0200 Received: from [10.20.13.30] (unknown [10.20.13.30]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by data.antarean.org (Postfix) with ESMTPSA id 97CAA4C for ; Mon, 4 Aug 2014 21:46:41 +0200 (CEST) User-Agent: K-9 Mail for Android In-Reply-To: <53DF8C2D.5070402@gmail.com> References: <53DBCF34.6060601@gmail.com> <10589ff2-a642-4951-955d-339d475ccaad@email.android.com> <4871526.Mj2HT7lMQH@andromeda> <53DF8C2D.5070402@gmail.com> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Subject: Re: [gentoo-user] Re: Recommendations for scheduler From: "J. Roeleveld" Date: Mon, 04 Aug 2014 21:46:39 +0200 To: gentoo-user@lists.gentoo.org Message-ID: <506301c4-0106-4ee6-b532-12d08b7a1ce1@email.android.com> Content-Transfer-Encoding: quoted-printable X-Ziggo-spambar: ---- X-Ziggo-spamscore: -4.9 X-Ziggo-spamreport: ALL_TRUSTED=-1,BAYES_00=-1.9,PROLO_TRUST_RDNS=-3,RDNS_DYNAMIC=0.982 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Archives-Salt: 08ec96f1-e702-4e3d-ae10-aa430dfabdee X-Archives-Hash: a54ffba5866b50ac1f31ad33591eb8f1 On 4 August 2014 15:35:41 CEST, Alan McKinnon w= rote: >On 04/08/2014 15:31, Martin Vaeth wrote: >> J. Roeleveld wrote: >>>> >>>> So you have a command which might break due to hardware error >>>> and cannot be rerun. I cannot see how any general-purpose scheduler >>>> might help you here: You either need to be able to split your >command >>>> into several (sequential) commands or you need something adapted >>>> for your particular command. >>> >>> A general-purpose scheduler can work, as they do exist. >>=20 >> I doubt that they can solve your problem. >> Let me repeat: You have a single program which accesses the database >> in a complex way and somewhere in the course of accessing it, the >> machine (or program) crashes. >> No general-purpose program can recover from this: You need >> particular knowledge of the database and the program if you even >> want to have a *chance* to recover from such a situation. >> A program with such a particular knowledge can hardly be called >> "general-purpose". > > >Joost, > >Either make the ETL tool pick up where it stopped and continue as it is >the only that knows what it was doing and how far it got. Or, wrap the >entire script in a single transaction. Alan, That would be the ideal solution. However, a single transaction dealing with around 500,000,000 rows will g= et me shot by the DBAs :) (Never mind that the performance of this will be such that having it all = done by an office full of secretaries might be quicker.) Having the ETL process clever enough to be able to pick up from any point= requires a degree of forward thinking and planning that is never done in= real life. I would love to design it like that as it isn't too difficult. But I alwa= ys get brought into these projects when implementing these structures wil= l require a full rewrite and getting the original architects to admit the= ir design can't be made restartable without human intervention. At which point the business simply says it is acceptable to have people d= o a manual rollback and restart the schedules from wherever it went wrong= . I'm sure your wife has similar experiences as this is why these projects = are always late to deliver and over budget. -- Joost --=20 Sent from my Android device with K-9 Mail. Please excuse my brevity.