From m.pearce at gold.ac.uk Tue Oct 3 04:43:47 2006 From: m.pearce at gold.ac.uk (Marcus Pearce) Date: Tue, 3 Oct 2006 11:43:47 +0100 (BST) Subject: [CLSQL-Devel] Re: [CLSQL-Help] Still broken disable-sql-reader-syntax In-Reply-To: References: <7.0.1.0.0.20060302100516.0382a2d8@algo.be> <20060302.105200.85340752.hanche@math.ntnu.no> <7.0.1.0.0.20060302112207.038d7440@algo.be> <20060302.115456.236653576.hanche@math.ntnu.no> Message-ID: On Thu, 28 Sep 2006, GP lisper wrote: > On Thu, 02 Mar 2006 wrote: >> >> Just start up a fresh lisp, load only clsql and nothing else, and test >> it like this: >> >> SCRATCH> '[foo] >> [FOO] >> SCRATCH> (clsql:locally-enable-sql-reader-syntax) >> ; No value >> SCRATCH> '[foo] >> # >> SCRATCH> (clsql:locally-disable-sql-reader-syntax) >> ; No value >> SCRATCH> '[foo] >> Warning: Ignoring unmatched close parenthesis at file position 6. >> [FOO >> >> locally-disable-sql-reader-syntax really should restore the >> syntax for #\] as well. > > This is still broken for me in clsql-3.7.2, error is now: > > Error in KERNEL::UNBOUND-SYMBOL-ERROR-HANDLER: the variable ] is unbound > > cmucl-19b Right, the problem here is that the call to LOCALLY-DISABLE-SQL-READER-SYNTAX ends up making the following function call: (SET-MACRO-CHARACTER #\] NIL) but SET-MACRO-CHARACTER requires a function designator as its second argument[1] and "The consequences are undefined if a symbol is used as a function designator but it does not have a global definition as a function, or it has a global definition as a macro or a special form" [2]. SBCL (and presumably CMUCL) does something odd related to the non-terminating-ness of the character while CLISP, for example, signals an error. I think the right thing to do is to reimplement %{ENABLE,DISABLE}-SQL-READER-SYNTAX such that they operate on a copy of the current readtable - see attached patch against 3.7.3. With the patch applied, the problem described above goes away and the test suite runs without failure on SBCL 0.9.16, Linux/x86, Postgresql 8.0.8. Cheers, Marcus [1] http://www.lispworks.com/documentation/HyperSpec/Body/f_set_ma.htm [2] http://www.lispworks.com/documentation/HyperSpec/Body/26_glo_f.htm#function_designator -------------- next part -------------- --- syntax-old.lisp 2006-10-03 10:23:01.000000000 +0100 +++ syntax.lisp 2006-10-03 10:46:53.000000000 +0100 @@ -16,9 +16,7 @@ (in-package #:clsql-sys) -(defvar *original-reader-enter* nil) - -(defvar *original-reader-exit* nil) +(defvar *original-readtable* nil) (defvar *sql-macro-open-char* #\[) @@ -45,10 +43,9 @@ (%disable-sql-reader-syntax))) (defun %disable-sql-reader-syntax () - (when *original-reader-enter* - (set-macro-character *sql-macro-open-char* *original-reader-enter*)) - (when *original-reader-exit* - (set-macro-character *sql-macro-close-char* *original-reader-exit*)) + (when *original-readtable* + (setf *readtable* *original-readtable* + *original-readtable* nil)) (values)) @@ -70,12 +67,10 @@ (%enable-sql-reader-syntax))) (defun %enable-sql-reader-syntax () - (unless (eq (get-macro-character *sql-macro-open-char*) #'sql-reader-open) - (setf *original-reader-enter* (get-macro-character *sql-macro-open-char*)) - (set-macro-character *sql-macro-open-char* #'sql-reader-open)) - (unless (eq (get-macro-character *sql-macro-close-char*) - (get-macro-character #\))) - (setf *original-reader-exit* (get-macro-character *sql-macro-close-char*)) + (unless *original-readtable* + (setf *original-readtable* *readtable* + *readtable* (copy-readtable)) + (set-macro-character *sql-macro-open-char* #'sql-reader-open) (set-macro-character *sql-macro-close-char* (get-macro-character #\)))) (values)) From kevin at rosenberg.net Tue Oct 3 05:22:32 2006 From: kevin at rosenberg.net (Kevin Rosenberg) Date: Tue, 3 Oct 2006 05:22:32 -0600 Subject: [CLSQL-Devel] Re: [CLSQL-Help] Still broken disable-sql-reader-syntax In-Reply-To: References: <7.0.1.0.0.20060302100516.0382a2d8@algo.be> <20060302.105200.85340752.hanche@math.ntnu.no> <7.0.1.0.0.20060302112207.038d7440@algo.be> <20060302.115456.236653576.hanche@math.ntnu.no> Message-ID: <20061003112232.GA21617@rosenberg.net> Marcus Pearce wrote: > I think the right thing to do is to reimplement > %{ENABLE,DISABLE}-SQL-READER-SYNTAX such that they operate on a copy of > the current readtable - see attached patch against 3.7.3. Thanks, Marcus, applied and released as 3.7.4 -- Kevin Rosenberg kevin at hypershots.com From andy at onshored.com Mon Oct 16 12:33:03 2006 From: andy at onshored.com (Andrew Golding) Date: Mon, 16 Oct 2006 13:33:03 -0500 Subject: [CLSQL-Devel] CLSQL Postgresql additions Message-ID: <1161023583.6839.14.camel@kfreedom.onshored.com> Hello: I've noticed that the error-id slot for the condition sql-database-data-error while using Postgres does not actually contain the 5-letter SQL error code specified by http://www.postgresql.org/docs/8.1/interactive/errcodes-appendix.html (and apparently to some degree the SQL spec), but rather the integer status code for the result set. Since the status code for fatal error is always 7, the error-id slot will always be 7. Setting error-id to the SQL error code (which can be done by calling PQresultErrorField with the result set and the constant PG_DIAG_SQLSTATE (equal to ascii character code for capitol C, specified in postgres_ext.h) is accomplished within the three files attached to this message. Postgresql-api.lisp contains the new UFFI function definition for PQresultErrorField and constants defined for each possible value for the field code argument. Postgresql-package.lisp exports the new function along with the constant +PG-DIAG-SQLSTATE+ (the only constant we are interested in at the moment). Postgresql-sql.lisp is changed so that when the sql-database-data-error is generated as the result of a bad sql command, error-id is set to the error code instead of the result status code. These changes allow application code which handles errors to discern between a constraint violation causing an sql-database-data-error and something more sinister (such as a disk being full or the data center where the machine is hosted being on fire). Let me know if there are any questions. The following changes started with files from CLSQL 3.7.4 Regards Andrew Golding -------------- next part -------------- ;;;; -*- Mode: LISP; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;;; ************************************************************************* ;;;; FILE IDENTIFICATION ;;;; ;;;; Name: postgresql.cl ;;;; Purpose: Low-level PostgreSQL interface using UFFI ;;;; Programmers: Kevin M. Rosenberg based on ;;;; Original code by Pierre R. Mai ;;;; Date Started: Feb 2002 ;;;; ;;;; $Id: postgresql-api.lisp 10077 2004-10-01 17:02:20Z kevin $ ;;;; ;;;; This file, part of CLSQL, is Copyright (c) 2002 by Kevin M. Rosenberg ;;;; and Copyright (c) 1999-2001 by Pierre R. Mai ;;;; ;;;; CLSQL users are granted the rights to distribute and use this software ;;;; as governed by the terms of the Lisp Lesser GNU Public License ;;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. ;;;; ************************************************************************* (in-package #:postgresql) ;;;; This file implements as little of the FFI bindings to the ;;;; PostgreSQL client libraries as we could get away with. ;;;; Especially all the PostgreSQL-specific goodies aren't there, and ;;;; we just use void pointers where we can get away with it, which ;;;; thanks to the design of the PostgreSQL client libraries is pretty ;;;; much everywhere, in contrast to the MySQL client libraries for ;;;; example. ;;;; Type definitions ;;; Basic Types (uffi:def-foreign-type pgsql-oid :unsigned-int) (uffi:def-enum pgsql-conn-status-type (:connection-ok :connection-bad)) (uffi:def-enum pgsql-exec-status-type (:empty-query :command-ok :tuples-ok :copy-out :copy-in :bad-response :nonfatal-error :fatal-error)) (uffi:def-foreign-type pgsql-conn :pointer-void) (uffi:def-foreign-type pgsql-result :pointer-void) (uffi:def-type pgsql-conn-ptr :pointer-void) (uffi:def-enum pgsql-ftype ((:bytea 17) (:int2 21) (:int4 23) (:int8 20) (:float4 700) (:float8 701))) ;;(declaim (inline PQsetdbLogin)) ;; causes compile error in LW 4.2.0 (uffi:def-function ("PQsetdbLogin" PQsetdbLogin) ((pghost :cstring) (pgport :cstring) (pgoptions :cstring) (pgtty :cstring) (dbName :cstring) (login :cstring) (pwd :cstring)) :module "postgresql" :returning pgsql-conn) (declaim (inline PQfinish)) (uffi:def-function ("PQfinish" PQfinish) ((conn pgsql-conn)) :module "postgresql" :returning :void) (declaim (inline PQstatus)) (uffi:def-function ("PQstatus" PQstatus) ((conn pgsql-conn)) :module "postgresql" :returning pgsql-conn-status-type) (declaim (inline PQerrorMessage)) (uffi:def-function ("PQerrorMessage" PQerrorMessage) ((conn pgsql-conn)) :module "postgresql" :returning :cstring) (declaim (inline PQexec)) (uffi:def-function ("PQexec" PQexec) ((conn pgsql-conn) (query :cstring)) :module "postgresql" :returning pgsql-result) (declaim (inline PQresultStatus)) (uffi:def-function ("PQresultStatus" PQresultStatus) ((res pgsql-result)) :module "postgresql" :returning pgsql-exec-status-type) ; From postgres_ext.h ; #define PG_DIAG_SEVERITY 'S' ; #define PG_DIAG_SQLSTATE 'C' ; #define PG_DIAG_MESSAGE_PRIMARY 'M' ; #define PG_DIAG_MESSAGE_DETAIL 'D' ; #define PG_DIAG_MESSAGE_HINT 'H' ; #define PG_DIAG_STATEMENT_POSITION 'P' ; #define PG_DIAG_INTERNAL_POSITION 'p' ; #define PG_DIAG_INTERNAL_QUERY 'q' ; #define PG_DIAG_CONTEXT 'W' ; #define PG_DIAG_SOURCE_FILE 'F' ; #define PG_DIAG_SOURCE_LINE 'L' ; #define PG_DIAG_SOURCE_FUNCTION 'R' (defconstant +PG-DIAG-SEVERITY+ (char-code #\S)) (defconstant +PG-DIAG-SQLSTATE+ (char-code #\C)) (defconstant +PG-DIAG-MESSAGE-PRIMARY+ (char-code #\M)) (defconstant +PG-DIAG-MESSAGE-DETAIL+ (char-code #\D)) (defconstant +PG-DIAG-MESSAGE-HINT+ (char-code #\H)) (defconstant +PG-DIAG-STATEMENT-POSITION+ (char-code #\P)) (defconstant +PG-DIAG-INTERNAL-POSITION+ (char-code #\p)) (defconstant +PG-DIAG-INTERNAL-QUERY+ (char-code #\q)) (defconstant +PG-DIAG-CONTEXT+ (char-code #\W)) (defconstant +PG-DIAG-SOURCE-FILE+ (char-code #\F)) (defconstant +PG-DIAG-SOURCE-LINE+ (char-code #\L)) (defconstant +PG-DIAG-SOURCE-FUNCTION+ (char-code #\R)) ; PQresultErrorField can return diagnostic information about an error (declaim (inline PQresultErrorField)) (uffi:def-function ("PQresultErrorField" PQresultErrorField) ((res pgsql-result) (field-code :int)) :module "postgresql" :returning (* char)) (declaim (inline PQresultErrorMessage)) (uffi:def-function ("PQresultErrorMessage" PQresultErrorMessage) ((res pgsql-result)) :module "postgresql" :returning :cstring) (declaim (inline PQntuples)) (uffi:def-function ("PQntuples" PQntuples) ((res pgsql-result)) :module "postgresql" :returning :int) (declaim (inline PQnfields)) (uffi:def-function ("PQnfields" PQnfields) ((res pgsql-result)) :module "postgresql" :returning :int) (declaim (inline PQfname)) (uffi:def-function ("PQfname" PQfname) ((res pgsql-result) (field-num :int)) :module "postgresql" :returning :cstring) (declaim (inline PQfnumber)) (uffi:def-function ("PQfnumber" PQfnumber) ((res pgsql-result) (field-name :cstring)) :module "postgresql" :returning :int) (declaim (inline PQftype)) (uffi:def-function ("PQftype" PQftype) ((res pgsql-result) (field-num :int)) :module "postgresql" :returning pgsql-oid) (declaim (inline PQfsize)) (uffi:def-function ("PQfsize" PQfsize) ((res pgsql-result) (field-num :int)) :module "postgresql" :returning :short) (declaim (inline PQcmdStatus)) (uffi:def-function ("PQcmdStatus" PQcmdStatus) ((res pgsql-result)) :module "postgresql" :returning :cstring) (declaim (inline PQoidStatus)) (uffi:def-function ("PQoidStatus" PQoidStatus) ((res pgsql-result)) :module "postgresql" :returning :cstring) (declaim (inline PQcmdTuples)) (uffi:def-function ("PQcmdTuples" PQcmdTuples) ((res pgsql-result)) :module "postgresql" :returning :cstring) (declaim (inline PQgetvalue)) (uffi:def-function ("PQgetvalue" PQgetvalue) ((res pgsql-result) (tup-num :int) (field-num :int)) :module "postgresql" :returning (* :unsigned-char)) (declaim (inline PQgetlength)) (uffi:def-function ("PQgetlength" PQgetlength) ((res pgsql-result) (tup-num :int) (field-num :int)) :module "postgresql" :returning :int) (declaim (inline PQgetisnull)) (uffi:def-function ("PQgetisnull" PQgetisnull) ((res pgsql-result) (tup-num :int) (field-num :int)) :module "postgresql" :returning :int) (declaim (inline PQclear)) (uffi:def-function ("PQclear" PQclear) ((res pgsql-result)) :module "postgresql" :returning :void) (declaim (inline PQisBusy)) (uffi:def-function ("PQisBusy" PQisBusy) ((conn pgsql-conn)) :module "postgresql" :returning :int) ;;; Large objects support (MB) (defconstant +INV_ARCHIVE+ 65536) ; fe-lobj.c (defconstant +INV_WRITE+ 131072) (defconstant +INV_READ+ 262144) (declaim (inline lo-creat)) (uffi:def-function ("lo_creat" lo-create) ((conn pgsql-conn) (mode :int)) :module "postgresql" :returning pgsql-oid) (declaim (inline lo-open)) (uffi:def-function ("lo_open" lo-open) ((conn pgsql-conn) (oid pgsql-oid) (mode :int)) :module "postgresql" :returning :int) (declaim (inline lo-write)) (uffi:def-function ("lo_write" lo-write) ((conn pgsql-conn) (fd :int) (data :cstring) (size :int)) :module "postgresql" :returning :int) (declaim (inline lo-read)) (uffi:def-function ("lo_read" lo-read) ((conn pgsql-conn) (fd :int) (data (* :unsigned-char)) (size :int)) :module "postgresql" :returning :int) (declaim (inline lo-lseek)) (uffi:def-function ("lo_lseek" lo-lseek) ((conn pgsql-conn) (fd :int) (offset :int) (whence :int)) :module "postgresql" :returning :int) (declaim (inline lo-close)) (uffi:def-function ("lo_close" lo-close) ((conn pgsql-conn) (fd :int)) :module "postgresql" :returning :int) (declaim (inline lo-unlink)) (uffi:def-function ("lo_unlink" lo-unlink) ((conn pgsql-conn) (oid pgsql-oid)) :module "postgresql" :returning :int) -------------- next part -------------- ;;;; -*- Mode: LISP; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;;; ************************************************************************* ;;;; FILE IDENTIFICATION ;;;; ;;;; Name: postgresql-package.cl ;;;; Purpose: Package definition for low-level PostgreSQL interface ;;;; Programmer: Kevin M. Rosenberg ;;;; Date Started: Feb 2002 ;;;; ;;;; $Id: postgresql-package.lisp 7061 2003-09-07 06:34:45Z kevin $ ;;;; ;;;; This file, part of CLSQL, is Copyright (c) 2002 by Kevin M. Rosenberg ;;;; ;;;; CLSQL users are granted the rights to distribute and use this software ;;;; as governed by the terms of the Lisp Lesser GNU Public License ;;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. ;;;; ************************************************************************* (in-package #:cl-user) (defpackage #:postgresql (:nicknames #:pgsql) (:use #:cl #:clsql-uffi) (:export #:pgsql-oid #:pgsql-conn-status-type #:pgsql-conn-status-type#connection-ok #:pgsql-conn-status-type#connection-bad #:pgsql-exec-status-type #:pgsql-exec-status-type#empty-query #:pgsql-exec-status-type#command-ok #:pgsql-exec-status-type#tuples-ok #:pgsql-exec-status-type#copy-out #:pgsql-exec-status-type#copy-in #:pgsql-exec-status-type#bad-response #:pgsql-exec-status-type#nonfatal-error #:pgsql-exec-status-type#fatal-error #:pgsql-conn #:pgsql-result #:pgsql-ftype#bytea #:pgsql-ftype#int2 #:pgsql-ftype#int4 #:pgsql-ftype#int8 #:pgsql-ftype#float4 #:pgsql-ftype#float8 ;; Used by PQresultErrorField to get the sql error code #:+PG-DIAG-SQLSTATE+ ;; Functions #:PQsetdbLogin #:PQlogin #:PQfinish #:PQstatus #:PQerrorMessage #:PQexec #:PQresultStatus #:PQresultErrorField ; used to grab the SQLSTATE code from an error #:PQresultErrorMessage #:PQntuples #:PQnfields #:PQfname #:PQfnumber #:PQftype #:PQfsize #:PQcmdStatus #:PQoidStatus #:PQcmdTuples #:PQgetvalue #:PQgetlength #:PQgetisnull #:PQclear #:PQisBusy ;;Large Objects (Marc B) #:lo-create #:lo-open #:lo-write #:lo-read #:lo-lseek #:lo-close #:lo-unlink ) (:documentation "This is the low-level interface to PostgreSQL.")) -------------- next part -------------- ;;;; -*- Mode: LISP; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;;; ************************************************************************* ;;;; FILE IDENTIFICATION ;;;; ;;;; Name: postgresql-sql.lisp ;;;; Purpose: High-level PostgreSQL interface using UFFI ;;;; Date Started: Feb 2002 ;;;; ;;;; $Id: postgresql-sql.lisp 10969 2006-07-05 02:16:49Z kevin $ ;;;; ;;;; CLSQL users are granted the rights to distribute and use this software ;;;; as governed by the terms of the Lisp Lesser GNU Public License ;;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. ;;;; ************************************************************************* (in-package #:cl-user) (defpackage #:clsql-postgresql (:use #:common-lisp #:clsql-sys #:postgresql #:clsql-uffi) (:export #:postgresql-database) (:documentation "This is the CLSQL interface to PostgreSQL.")) (in-package #:clsql-postgresql) ;;; Field conversion functions (defun make-type-list-for-auto (num-fields res-ptr) (let ((new-types '())) (dotimes (i num-fields) (declare (fixnum i)) (let* ((type (PQftype res-ptr i))) (push (case type ((#.pgsql-ftype#bytea #.pgsql-ftype#int2 #.pgsql-ftype#int4) :int32) (#.pgsql-ftype#int8 :int64) ((#.pgsql-ftype#float4 #.pgsql-ftype#float8) :double) (otherwise t)) new-types))) (nreverse new-types))) (defun canonicalize-types (types num-fields res-ptr) (if (null types) nil (let ((auto-list (make-type-list-for-auto num-fields res-ptr))) (cond ((listp types) (canonicalize-type-list types auto-list)) ((eq types :auto) auto-list) (t nil))))) (defun tidy-error-message (message) (unless (stringp message) (setq message (uffi:convert-from-foreign-string message))) (let ((message (string-right-trim '(#\Return #\Newline) message))) (cond ((< (length message) (length "ERROR:")) message) ((string= message "ERROR:" :end1 6) (string-left-trim '(#\Space) (subseq message 6))) (t message)))) (defmethod database-initialize-database-type ((database-type (eql :postgresql))) t) (uffi:def-type pgsql-conn-def pgsql-conn) (uffi:def-type pgsql-result-def pgsql-result) (defclass postgresql-database (generic-postgresql-database) ((conn-ptr :accessor database-conn-ptr :initarg :conn-ptr :type pgsql-conn-def) (lock :accessor database-lock :initform (make-process-lock "conn")))) (defmethod database-type ((database postgresql-database)) :postgresql) (defmethod database-name-from-spec (connection-spec (database-type (eql :postgresql))) (check-connection-spec connection-spec database-type (host db user password &optional port options tty)) (destructuring-bind (host db user password &optional port options tty) connection-spec (declare (ignore password options tty)) (concatenate 'string (etypecase host (null "localhost") (pathname (namestring host)) (string host)) (when port (concatenate 'string ":" (etypecase port (integer (write-to-string port)) (string port)))) "/" db "/" user))) (defmethod database-connect (connection-spec (database-type (eql :postgresql))) (check-connection-spec connection-spec database-type (host db user password &optional port options tty)) (destructuring-bind (host db user password &optional port options tty) connection-spec (uffi:with-cstrings ((host-native host) (user-native user) (password-native password) (db-native db) (port-native port) (options-native options) (tty-native tty)) (let ((connection (PQsetdbLogin host-native port-native options-native tty-native db-native user-native password-native))) (declare (type pgsql-conn-def connection)) (when (not (eq (PQstatus connection) pgsql-conn-status-type#connection-ok)) (let ((pqstatus (PQstatus connection)) (pqmessage (tidy-error-message (PQerrorMessage connection)))) (PQfinish connection) (error 'sql-connection-error :database-type database-type :connection-spec connection-spec :error-id pqstatus :message pqmessage))) (make-instance 'postgresql-database :name (database-name-from-spec connection-spec database-type) :database-type :postgresql :connection-spec connection-spec :conn-ptr connection))))) (defmethod database-disconnect ((database postgresql-database)) (PQfinish (database-conn-ptr database)) (setf (database-conn-ptr database) nil) t) (defmethod database-query (query-expression (database postgresql-database) result-types field-names) (let ((conn-ptr (database-conn-ptr database))) (declare (type pgsql-conn-def conn-ptr)) (uffi:with-cstring (query-native query-expression) (let ((result (PQexec conn-ptr query-native))) (when (uffi:null-pointer-p result) (error 'sql-database-data-error :database database :expression query-expression :message (tidy-error-message (PQerrorMessage conn-ptr)))) (unwind-protect (case (PQresultStatus result) ;; User gave a command rather than a query (#.pgsql-exec-status-type#command-ok nil) (#.pgsql-exec-status-type#empty-query nil) (#.pgsql-exec-status-type#tuples-ok (let ((num-fields (PQnfields result))) (when result-types (setq result-types (canonicalize-types result-types num-fields result))) (let ((res (loop for tuple-index from 0 below (PQntuples result) collect (loop for i from 0 below num-fields collect (if (zerop (PQgetisnull result tuple-index i)) (convert-raw-field (PQgetvalue result tuple-index i) result-types i) nil))))) (if field-names (values res (result-field-names num-fields result)) res)))) (t (error 'sql-database-data-error :database database :expression query-expression :error-id (PQresultStatus result) :message (tidy-error-message (PQresultErrorMessage result))))) (PQclear result)))))) (defun result-field-names (num-fields result) "Return list of result field names." (let ((names '())) (dotimes (i num-fields (nreverse names)) (declare (fixnum i)) (push (uffi:convert-from-cstring (PQfname result i)) names)))) (defmethod database-execute-command (sql-expression (database postgresql-database)) (let ((conn-ptr (database-conn-ptr database))) (declare (type pgsql-conn-def conn-ptr)) (uffi:with-cstring (sql-native sql-expression) (let ((result (PQexec conn-ptr sql-native))) (when (uffi:null-pointer-p result) (error 'sql-database-data-error :database database :expression sql-expression :message (tidy-error-message (PQerrorMessage conn-ptr)))) (unwind-protect (case (PQresultStatus result) (#.pgsql-exec-status-type#command-ok t) ((#.pgsql-exec-status-type#empty-query #.pgsql-exec-status-type#tuples-ok) (warn "Strange result...") t) (t (error 'sql-database-data-error :database database :expression sql-expression :error-id (uffi:convert-from-foreign-string (PQresultErrorField result +PG-DIAG-SQLSTATE+)) :message (tidy-error-message (PQresultErrorMessage result))))) (PQclear result)))))) (defstruct postgresql-result-set (res-ptr (uffi:make-null-pointer 'pgsql-result) :type pgsql-result-def) (types nil) (num-tuples 0 :type integer) (num-fields 0 :type integer) (tuple-index 0 :type integer)) (defmethod database-query-result-set ((query-expression string) (database postgresql-database) &key full-set result-types) (let ((conn-ptr (database-conn-ptr database))) (declare (type pgsql-conn-def conn-ptr)) (uffi:with-cstring (query-native query-expression) (let ((result (PQexec conn-ptr query-native))) (when (uffi:null-pointer-p result) (error 'sql-database-data-error :database database :expression query-expression :message (tidy-error-message (PQerrorMessage conn-ptr)))) (case (PQresultStatus result) ((#.pgsql-exec-status-type#empty-query #.pgsql-exec-status-type#tuples-ok) (let ((result-set (make-postgresql-result-set :res-ptr result :num-fields (PQnfields result) :num-tuples (PQntuples result) :types (canonicalize-types result-types (PQnfields result) result)))) (if full-set (values result-set (PQnfields result) (PQntuples result)) (values result-set (PQnfields result))))) (t (unwind-protect (error 'sql-database-data-error :database database :expression query-expression :error-id (PQresultStatus result) :message (tidy-error-message (PQresultErrorMessage result))) (PQclear result)))))))) (defmethod database-dump-result-set (result-set (database postgresql-database)) (let ((res-ptr (postgresql-result-set-res-ptr result-set))) (declare (type pgsql-result-def res-ptr)) (PQclear res-ptr) t)) (defmethod database-store-next-row (result-set (database postgresql-database) list) (let ((result (postgresql-result-set-res-ptr result-set)) (types (postgresql-result-set-types result-set))) (declare (type pgsql-result-def result)) (if (>= (postgresql-result-set-tuple-index result-set) (postgresql-result-set-num-tuples result-set)) nil (loop with tuple-index = (postgresql-result-set-tuple-index result-set) for i from 0 below (postgresql-result-set-num-fields result-set) for rest on list do (setf (car rest) (if (zerop (PQgetisnull result tuple-index i)) (convert-raw-field (PQgetvalue result tuple-index i) types i) nil)) finally (incf (postgresql-result-set-tuple-index result-set)) (return list))))) ;;; Large objects support (Marc B) (defmethod database-create-large-object ((database postgresql-database)) (lo-create (database-conn-ptr database) (logior postgresql::+INV_WRITE+ postgresql::+INV_READ+))) #+mb-original (defmethod database-write-large-object (object-id (data string) (database postgresql-database)) (let ((ptr (database-conn-ptr database)) (length (length data)) (result nil) (fd nil)) (with-transaction (:database database) (unwind-protect (progn (setf fd (lo-open ptr object-id postgresql::+INV_WRITE+)) (when (>= fd 0) (when (= (lo-write ptr fd data length) length) (setf result t)))) (progn (when (and fd (>= fd 0)) (lo-close ptr fd)) ))) result)) (defmethod database-write-large-object (object-id (data string) (database postgresql-database)) (let ((ptr (database-conn-ptr database)) (length (length data)) (result nil) (fd nil)) (database-execute-command "begin" database) (unwind-protect (progn (setf fd (lo-open ptr object-id postgresql::+INV_WRITE+)) (when (>= fd 0) (when (= (lo-write ptr fd data length) length) (setf result t)))) (progn (when (and fd (>= fd 0)) (lo-close ptr fd)) (database-execute-command (if result "commit" "rollback") database))) result)) ;; (MB) the begin/commit/rollback stuff will be removed when with-transaction wil be implemented ;; (KMR) Can't use with-transaction since that function is in high-level code (defmethod database-read-large-object (object-id (database postgresql-database)) (let ((ptr (database-conn-ptr database)) (buffer nil) (result nil) (length 0) (fd nil)) (unwind-protect (progn (database-execute-command "begin" database) (setf fd (lo-open ptr object-id postgresql::+INV_READ+)) (when (>= fd 0) (setf length (lo-lseek ptr fd 0 2)) (lo-lseek ptr fd 0 0) (when (> length 0) (setf buffer (uffi:allocate-foreign-string length :unsigned t)) (when (= (lo-read ptr fd buffer length) length) (setf result (uffi:convert-from-foreign-string buffer :length length :null-terminated-p nil)))))) (progn (when buffer (uffi:free-foreign-object buffer)) (when (and fd (>= fd 0)) (lo-close ptr fd)) (database-execute-command (if result "commit" "rollback") database))) result)) (defmethod database-delete-large-object (object-id (database postgresql-database)) (lo-unlink (database-conn-ptr database) object-id)) ;;; Object listing (defmethod database-create (connection-spec (type (eql :postgresql))) (destructuring-bind (host name user password) connection-spec (declare (ignore user password)) (multiple-value-bind (output status) (clsql-sys:command-output "createdb -h~A ~A" (if host host "localhost") name) (if (or (not (zerop status)) (search "database creation failed: ERROR:" output)) (error 'sql-database-error :message (format nil "createdb failed for postgresql backend with connection spec ~A." connection-spec)) t)))) (defmethod database-destroy (connection-spec (type (eql :postgresql))) (destructuring-bind (host name user password) connection-spec (declare (ignore user password)) (multiple-value-bind (output status) (clsql-sys:command-output "dropdb -h~A ~A" (if host host "localhost") name) (if (or (not (zerop status)) (search "database removal failed: ERROR:" output)) (error 'sql-database-error :message (format nil "dropdb failed for postgresql backend with connection spec ~A." connection-spec)) t)))) (defmethod database-probe (connection-spec (type (eql :postgresql))) (when (find (second connection-spec) (database-list connection-spec type) :test #'string-equal) t)) (defun %pg-database-connection (connection-spec) (check-connection-spec connection-spec :postgresql (host db user password &optional port options tty)) (macrolet ((coerce-string (var) `(unless (typep ,var 'simple-base-string) (setf ,var (coerce ,var 'simple-base-string))))) (destructuring-bind (host db user password &optional port options tty) connection-spec (coerce-string db) (coerce-string user) (let ((connection (PQsetdbLogin host port options tty db user password))) (declare (type postgresql::pgsql-conn-ptr connection)) (unless (eq (PQstatus connection) pgsql-conn-status-type#connection-ok) ;; Connect failed (error 'sql-connection-error :database-type :postgresql :connection-spec connection-spec :error-id (PQstatus connection) :message (PQerrorMessage connection))) connection)))) (defmethod database-reconnect ((database postgresql-database)) (let ((lock (database-lock database))) (with-process-lock (lock "Reconnecting") (with-slots (connection-spec conn-ptr) database (setf conn-ptr (%pg-database-connection connection-spec)) database)))) ;;; Database capabilities (when (clsql-sys:database-type-library-loaded :postgresql) (clsql-sys:initialize-database-type :database-type :postgresql)) From kevin at rosenberg.net Mon Oct 16 13:07:12 2006 From: kevin at rosenberg.net (Kevin Rosenberg) Date: Mon, 16 Oct 2006 13:07:12 -0600 Subject: [CLSQL-Devel] CLSQL Postgresql additions In-Reply-To: <1161023583.6839.14.camel@kfreedom.onshored.com> References: <1161023583.6839.14.camel@kfreedom.onshored.com> Message-ID: <20061016190712.GA13836@rosenberg.net> Andrew Golding wrote: > I've noticed that the error-id slot for the condition > sql-database-data-error while using Postgres does not actually contain > the 5-letter SQL error code specified by > [...] Thanks for the notes and changes, Andrew and welcome to the changelog ;-) While your changes are most welcome, in the future, unified diffs are the preferred form for receiving patches. Your changes are incorporated in CLSQL 3.7.5 which will be released today. Thanks! Kevin From jsb at nil.at Wed Oct 18 07:41:43 2006 From: jsb at nil.at (Hans Bulfone) Date: Wed, 18 Oct 2006 15:41:43 +0200 Subject: [CLSQL-Devel] getting the number of affected rows Message-ID: <20061018134143.GB23861@atnet.at> hi, i need a way to get the number of affected rows of an sql update or delete expression, but it seems this is not supported by the current clsql version. this could be easily implemented, at least for postgresql and mysql, by changing database-execute-command to return the number of affected rows (got via PQcmdTuples or mysql_affected_rows) and execute-command to return the result of database-execute-command. what do you think of this? i can write a patch with the proposed changes for postgresql and mysql, (and perhaps sqlite2/3). bye, hans. From kevin at rosenberg.net Wed Oct 18 21:54:25 2006 From: kevin at rosenberg.net (Kevin Rosenberg) Date: Wed, 18 Oct 2006 21:54:25 -0600 Subject: [CLSQL-Devel] getting the number of affected rows In-Reply-To: <20061018134143.GB23861@atnet.at> References: <20061018134143.GB23861@atnet.at> Message-ID: <20061019035425.GA21569@rosenberg.net> Hans Bulfone wrote: > i need a way to get the number of affected rows of an > sql update or delete expression, but it seems this is > not supported by the current clsql version. The number of affector rows can be handy. Unfortunately, the SQL specification does not include that ability, thus CommonSQL does not include that ability: http://www.lispworks.com/documentation/lw445/LWRM/html/lwref-460.htm > this could be easily implemented, at least for postgresql and mysql, > by changing database-execute-command to return the number > of affected rows (got via PQcmdTuples or mysql_affected_rows) > and execute-command to return the result of database-execute-command. Yes, some SQL implementations do provide that ability since it can be useful information. I suspect the standard SQL approach would be to lock the table, perform a SELECT count(*) matching the rows you'll be modifying, perform your EXECUTE-COMMAND, then unlock the table. Obviously, if commands could return counts in SQL, then that SELECT step could be avoided. > what do you think of this? Pros: - more efficient than the above SELECT first approach above Cons: - not backwards compatible with CommonSQL (not a huge problem) - can be worked around by above SELECT method [relatively inefficent] or by using CLSQL backend internals like (mysql:mysql_affected_rows (clsql-sys::database-mysql-ptr db)) [not portable across backends, depends on undocumented features] - not known to be able to supported by all supported backends. One of the advantages (and goals) of CLSQL is try to have neutrality toward sql backends. Obviously, this isn't completely provided. However, adding a fundemental feature to EXECUTE-COMMAND should be able to be supported by ideally all CLSQL backends. db-aodbc (not a popular backend) likely can't support this. However, db-aodbc can not support all CLSQL as it. ODBC and Oracle are more popular backends. I think those backends would need to support this if extension were to be added, documented, and offically supported. > i can write a patch with the proposed changes for postgresql and mysql, > (and perhaps sqlite2/3). Very good. For mysql, it's a one line addition (as above). What do you think about adding support ODBC and Oracle in addition to the backends you mentioned? Also, additions would need to be made to the test suite to verify proper working of this over time. The update to the documentation would be quite simple. I realize this task may seem larger than you originally thought. You might prefer to use one of the alternative approaches above rather than working on a robust addition to CLSQL. Kevin From jsb at nil.at Thu Oct 19 16:52:16 2006 From: jsb at nil.at (Hans Bulfone) Date: Fri, 20 Oct 2006 00:52:16 +0200 Subject: [CLSQL-Devel] getting the number of affected rows In-Reply-To: <20061019035425.GA21569@rosenberg.net> References: <20061018134143.GB23861@atnet.at> <20061019035425.GA21569@rosenberg.net> Message-ID: <20061019225215.GA2160@atnet.at> hi, On Wed, Oct 18, 2006 at 09:54:25PM -0600, Kevin Rosenberg wrote: > > i can write a patch with the proposed changes for postgresql and mysql, > > (and perhaps sqlite2/3). > > Very good. For mysql, it's a one line addition (as above). What do you > think about adding support ODBC and Oracle in addition to the backends > you mentioned? from looking at the code it seems that the odbc backend already supports this, i will try it tomorrow. with oracle, it looks like one can use oci-attr-get to get the +oci-attr-row-count+ of the statement handle. i can add that to my patch, but cannot easily test it as i don't have access to an oracle database. > Also, additions would need to be made to the test suite to verify > proper working of this over time. The update to the documentation > would be quite simple. > > I realize this task may seem larger than you originally thought. You > might prefer to use one of the alternative approaches above rather > than working on a robust addition to CLSQL. unfortunately, with postgresql you need access to the result object which is only available inside database-execute-command and i don't really like the table locking approach. also, i'd be glad to help improve clsql :) bye, hans. From kevin at rosenberg.net Fri Oct 20 08:58:35 2006 From: kevin at rosenberg.net (Kevin Rosenberg) Date: Fri, 20 Oct 2006 08:58:35 -0600 Subject: [CLSQL-Devel] getting the number of affected rows In-Reply-To: <20061019225215.GA2160@atnet.at> References: <20061018134143.GB23861@atnet.at> <20061019035425.GA21569@rosenberg.net> <20061019225215.GA2160@atnet.at> Message-ID: <20061020145835.GB30220@rosenberg.net> Hans Bulfone wrote: > from looking at the code it seems that the odbc backend already > supports this, i will try it tomorrow. Very good. > with oracle, it looks like one can use oci-attr-get to get the > +oci-attr-row-count+ of the statement handle. > i can add that to my patch, but cannot easily test it as i don't > have access to an oracle database. Understood. I don't have one either, but I know a few people who do. > unfortunately, with postgresql you need access to the result object > which is only available inside database-execute-command and i don't > really like the table locking approach. Makes sense. > also, i'd be glad to help improve clsql :) Excellent, appreciate the assistance in improving CLSQL! -- Kevin Rosenberg kevin at hypershots.com From jsb at nil.at Sun Oct 22 17:40:31 2006 From: jsb at nil.at (Hans Bulfone) Date: Mon, 23 Oct 2006 01:40:31 +0200 Subject: [CLSQL-Devel] getting the number of affected rows In-Reply-To: <20061020145835.GB30220@rosenberg.net> References: <20061018134143.GB23861@atnet.at> <20061019035425.GA21569@rosenberg.net> <20061019225215.GA2160@atnet.at> <20061020145835.GB30220@rosenberg.net> Message-ID: <20061022234029.GA2255@atnet.at> hi, the patch is now ready. it includes support for all backends except aodbc and db2. i haven't tested oracle. for odbc there was nothing to do. there were a few problems were i'm unsure if i picked the best solution: 1. in the :fdml/transaction/3 test the result of update-records is part of the test result. in my patch they are simply removed from the test result as it seems there is no easy way to make the test result depend on the database backend. (except perhaps putting (eql (update-records ...) (dbtype-dependent-expected-value)) in the test). the patch includes a new test for the rowcount feature which is skipped on aodbc. 2. the change in sql/utils.lisp is not related to the rest of the patch but was required in my case to get the test suite working with sqlite2 and sqlite3 below 3.2.4. the problem is that on sbcl (write-to-string 8e8 :readably t) => "8.e8" which is not a valid floating point literal in those sqlite versions. on clisp (write-to-string 8e8 :readably t) => "8.0f8" which is fine. i don't know about other cl implementations. bye, hans. -------------- next part -------------- diff -Nru clsql-3.7.7.ORIG/db-mysql/mysql-sql.lisp clsql-3.7.7/db-mysql/mysql-sql.lisp --- clsql-3.7.7.ORIG/db-mysql/mysql-sql.lisp 2005-11-10 23:03:45.000000000 +0100 +++ clsql-3.7.7/db-mysql/mysql-sql.lisp 2006-10-21 18:11:40.000000000 +0200 @@ -226,7 +226,7 @@ (declare (type mysql-mysql-ptr-def mysql-ptr)) (if (zerop (mysql-real-query mysql-ptr sql-native (expression-length sql-expression))) - t + (mysql-affected-rows mysql-ptr) (error 'sql-database-data-error :database database :expression sql-expression diff -Nru clsql-3.7.7.ORIG/db-oracle/oracle-sql.lisp clsql-3.7.7/db-oracle/oracle-sql.lisp --- clsql-3.7.7.ORIG/db-oracle/oracle-sql.lisp 2006-05-10 02:45:15.000000000 +0200 +++ clsql-3.7.7/db-oracle/oracle-sql.lisp 2006-10-22 18:24:46.000000000 +0200 @@ -502,7 +502,7 @@ (with-slots (envhp svchp errhp) db (uffi:with-foreign-strings ((c-stmt-string sql-stmt-string)) (let ((stmthp (uffi:allocate-foreign-object :pointer-void)) - select-p) + select-p changed-rows) (uffi:with-foreign-object (stmttype :unsigned-short) (unwind-protect @@ -530,7 +530,17 @@ (deref-vp stmthp) (deref-vp errhp) iters 0 +null-void-pointer+ +null-void-pointer+ +oci-default+ - :database db))) + :database db)) + (unless select-p + (uffi:with-foreign-object (rowcount 'ub4) + (oci-attr-get (deref-vp stmthp) + +oci-htype-stmt+ + rowcount + +unsigned-int-null-pointer+ + +oci-attr-row-count+ + (deref-vp errhp) + :database db) + (setq changed-rows (uffi:deref-pointer rowcount 'ub4))))) ;; free resources unless a query (unless select-p (oci-handle-free (deref-vp stmthp) +oci-htype-stmt+) @@ -540,7 +550,7 @@ (select-p (make-query-cursor db stmthp result-types field-names)) (t - nil)))))) + changed-rows)))))) ;; Return a QUERY-CURSOR representing the table returned from the OCI @@ -892,8 +902,8 @@ (defmethod database-query (query-expression (database oracle-database) result-types field-names) (let ((cursor (sql-stmt-exec query-expression database result-types field-names))) ;; (declare (type (or query-cursor null) cursor)) - (if (null cursor) ; No table was returned. - (values) + (if (or (null cursor) (numberp cursor)) ; No table was returned. + cursor (do ((reversed-result nil)) (nil) (let* ((eof-value :eof) @@ -958,10 +968,10 @@ (mapcar #'car (database-query query database nil nil)))) (defmethod database-execute-command (sql-expression (database oracle-database)) - (database-query sql-expression database nil nil) - (when (database-autocommit database) - (oracle-commit database)) - t) + (prog1 + (database-query sql-expression database nil nil) + (when (database-autocommit database) + (oracle-commit database)))) (defstruct (cd (:constructor make-cd) diff -Nru clsql-3.7.7.ORIG/db-postgresql/postgresql-sql.lisp clsql-3.7.7/db-postgresql/postgresql-sql.lisp --- clsql-3.7.7.ORIG/db-postgresql/postgresql-sql.lisp 2006-10-16 21:38:09.000000000 +0200 +++ clsql-3.7.7/db-postgresql/postgresql-sql.lisp 2006-10-18 16:37:37.000000000 +0200 @@ -213,7 +213,7 @@ (unwind-protect (case (PQresultStatus result) (#.pgsql-exec-status-type#command-ok - t) + (parse-integer (uffi:convert-from-cstring (PQcmdTuples result)) :junk-allowed t)) ((#.pgsql-exec-status-type#empty-query #.pgsql-exec-status-type#tuples-ok) (warn "Strange result...") diff -Nru clsql-3.7.7.ORIG/db-postgresql-socket/postgresql-socket-sql.lisp clsql-3.7.7/db-postgresql-socket/postgresql-socket-sql.lisp --- clsql-3.7.7.ORIG/db-postgresql-socket/postgresql-socket-sql.lisp 2005-09-18 02:13:11.000000000 +0200 +++ clsql-3.7.7/db-postgresql-socket/postgresql-socket-sql.lisp 2006-10-21 19:00:15.000000000 +0200 @@ -102,6 +102,21 @@ new-types))) (nreverse new-types))) +;; Result message parsing +(defun parse-command-result (result) + (let ((i (min 7 (length result)))) + (cond + ((string= result "INSERT " :end1 i) + (multiple-value-bind (oid pos) + (parse-integer result :start 7 :junk-allowed t) + (declare (ignore oid)) + (values (parse-integer result :start (1+ pos) :junk-allowed t)))) + ((string= result "UPDATE " :end1 i) + (values (parse-integer result :start 7 :junk-allowed t))) + ((string= result "DELETE " :end1 i) + (values (parse-integer result :start 7 :junk-allowed t))) + (t nil)))) + (defun convert-to-clsql-warning (database condition) (ecase *backend-warning-behavior* @@ -264,7 +279,7 @@ :expression expression :error-id "multiple-results" :message "Received multiple results for command.")) - result) + (parse-command-result result)) (t (close-postgresql-connection connection) (error 'sql-database-data-error diff -Nru clsql-3.7.7.ORIG/db-sqlite/sqlite-api.lisp clsql-3.7.7/db-sqlite/sqlite-api.lisp --- clsql-3.7.7.ORIG/db-sqlite/sqlite-api.lisp 2004-10-03 17:45:47.000000000 +0200 +++ clsql-3.7.7/db-sqlite/sqlite-api.lisp 2006-10-22 18:10:21.000000000 +0200 @@ -42,6 +42,7 @@ #:sqlite-version ; Defined as constant. #:sqlite-encoding ; Defined as constant. #:sqlite-last-insert-rowid + #:sqlite-changes ;;; Utility functions. #:make-null-row @@ -185,6 +186,12 @@ ((db sqlite-db)) :returning :int) +(declaim (inline sqlite-changes)) +(def-sqlite-function + "sqlite_changes" + ((db sqlite-db)) + :returning :int) + (declaim (inline %get-table)) (def-sqlite-function ("sqlite_get_table" %get-table) diff -Nru clsql-3.7.7.ORIG/db-sqlite/sqlite-sql.lisp clsql-3.7.7/db-sqlite/sqlite-sql.lisp --- clsql-3.7.7.ORIG/db-sqlite/sqlite-sql.lisp 2005-01-23 06:28:03.000000000 +0100 +++ clsql-3.7.7/db-sqlite/sqlite-sql.lisp 2006-10-22 18:12:07.000000000 +0200 @@ -72,7 +72,7 @@ :expression sql-expression :error-id (sqlite:sqlite-error-code err) :message (sqlite:sqlite-error-message err)))) - t) + (sqlite:sqlite-changes (sqlite-db database))) (defstruct sqlite-result-set (vm (sqlite:make-null-vm) diff -Nru clsql-3.7.7.ORIG/db-sqlite3/sqlite3-api.lisp clsql-3.7.7/db-sqlite3/sqlite3-api.lisp --- clsql-3.7.7.ORIG/db-sqlite3/sqlite3-api.lisp 2004-12-07 07:11:55.000000000 +0100 +++ clsql-3.7.7/db-sqlite3/sqlite3-api.lisp 2006-10-22 17:01:35.000000000 +0200 @@ -33,6 +33,7 @@ #:sqlite3-prepare #:sqlite3-step #:sqlite3-finalize + #:sqlite3-changes #:sqlite3-column-count #:sqlite3-column-name @@ -245,6 +246,12 @@ ((stmt sqlite3-stmt)) :returning :int) +(declaim (inline sqlite3-changes)) +(def-sqlite3-function + "sqlite3_changes" + ((db sqlite3-db)) + :returning :int) + (declaim (inline sqlite3-column-count)) (def-sqlite3-function "sqlite3_column_count" diff -Nru clsql-3.7.7.ORIG/db-sqlite3/sqlite3-sql.lisp clsql-3.7.7/db-sqlite3/sqlite3-sql.lisp --- clsql-3.7.7.ORIG/db-sqlite3/sqlite3-sql.lisp 2005-01-23 06:28:03.000000000 +0100 +++ clsql-3.7.7/db-sqlite3/sqlite3-sql.lisp 2006-10-22 17:09:16.000000000 +0200 @@ -79,7 +79,7 @@ :expression sql-expression :error-id (sqlite3:sqlite3-error-code err) :message (sqlite3:sqlite3-error-message err)))) - t) + (sqlite3:sqlite3-changes (sqlite3-db database))) (defstruct sqlite3-result-set (stmt sqlite3:null-stmt diff -Nru clsql-3.7.7.ORIG/doc/ref-fdml.xml clsql-3.7.7/doc/ref-fdml.xml --- clsql-3.7.7.ORIG/doc/ref-fdml.xml 2006-08-30 13:08:18.000000000 +0200 +++ clsql-3.7.7/doc/ref-fdml.xml 2006-10-23 00:24:40.423911716 +0200 @@ -232,7 +232,7 @@ Syntax - insert-records &key into attributes values av-pairs query database => + insert-records &key into attributes values av-pairs query database => rowcount Arguments and Values @@ -287,7 +287,13 @@ object. This will default to the value of *default-database*. - + + + rowcount + + The number of rows inserted. + + @@ -339,7 +345,7 @@ ecompanyid managerid) :values '(11 1 "Yuri" "Gagarin" "gagarin at soviet.org" 1 1)) -=> +=> 1 (select [first-name] [last-name] [email] :from [employee] :where [= [emplid] 11] @@ -395,7 +401,7 @@ Syntax - update-records table &key attributes values av-pairs where database => + update-records table &key attributes values av-pairs where database => rowcount Arguments and Values @@ -450,7 +456,13 @@ object. This will default to the value of *default-database*. - + + + rowcount + + The number of rows updated. + + @@ -492,7 +504,7 @@ (last_name "Gagarin") (email "gagarin at soviet.org")) :where [= [emplid] 1]) -=> +=> 1 (select [first-name] [last-name] [email] :from [employee] :where [= [emplid] 1] @@ -555,7 +567,7 @@ Syntax - delete-records &key from where database => + delete-records &key from where database => rowcount Arguments and Values @@ -587,6 +599,12 @@ of *default-database*. + + rowcount + + The number of rows deleted. + + @@ -607,7 +625,7 @@ :field-names nil) => (("Yuri" "Gagarin" "gagarin at soviet.org")) (delete-records :from [employee] :where [= [emplid] 11]) -=> +=> 1 (select [first-name] [last-name] [email] :from [employee] :where [= [emplid] 11] @@ -669,7 +687,7 @@ Syntax - execute-command sql-expression &key database => + execute-command sql-expression &key database => rowcount Arguments and Values @@ -690,6 +708,12 @@ object. This will default to the value of *default-database*. + + + rowcount + + The number of rows affected by the query. + diff -Nru clsql-3.7.7.ORIG/sql/fddl.lisp clsql-3.7.7/sql/fddl.lisp --- clsql-3.7.7.ORIG/sql/fddl.lisp 2005-11-26 17:08:12.000000000 +0100 +++ clsql-3.7.7/sql/fddl.lisp 2006-10-23 00:28:20.282001885 +0200 @@ -76,7 +76,8 @@ :columns description :modifiers constraints :transactions transactions))) - (execute-command stmt :database database))) + (execute-command stmt :database database) + (values))) (defun drop-table (name &key (if-does-not-exist :error) (database *default-database*) @@ -103,7 +104,8 @@ (symbol-name '#:clsql-oracle))))) (setq expr (concatenate 'string expr " PURGE"))) - (execute-command expr :database database)))) + (execute-command expr :database database) + (values)))) (defun list-tables (&key (owner nil) (database *default-database*)) "Returns a list of strings representing table names in DATABASE @@ -145,7 +147,8 @@ :column-list column-list :query as :with-check-option with-check-option))) - (execute-command stmt :database database))) + (execute-command stmt :database database) + (values))) (defun drop-view (name &key (if-does-not-exist :error) (database *default-database*)) @@ -161,7 +164,8 @@ (:error t)) (let ((expr (concatenate 'string "DROP VIEW " view-name))) - (execute-command expr :database database)))) + (execute-command expr :database database) + (values)))) (defun list-views (&key (owner nil) (database *default-database*)) "Returns a list of strings representing view names in DATABASE @@ -199,7 +203,8 @@ (stmt (format nil "CREATE ~A INDEX ~A ON ~A (~{~A~^, ~})" (if unique "UNIQUE" "") index-name table-name attributes))) - (execute-command stmt :database database))) + (execute-command stmt :database database) + (values))) (defun drop-index (name &key (if-does-not-exist :error) (on nil) @@ -223,7 +228,8 @@ (format nil "~A ON ~A" index-name (database-identifier on database))) (t index-name)))) (execute-command (format nil "DROP INDEX ~A" index-identifier) - :database database)))) + :database database) + (values)))) (defun list-indexes (&key (owner nil) (database *default-database*) (on nil)) "Returns a list of strings representing index names in DATABASE diff -Nru clsql-3.7.7.ORIG/sql/fdml.lisp clsql-3.7.7/sql/fdml.lisp --- clsql-3.7.7.ORIG/sql/fdml.lisp 2004-10-19 20:40:27.000000000 +0200 +++ clsql-3.7.7/sql/fdml.lisp 2006-10-18 16:38:40.000000000 +0200 @@ -25,13 +25,12 @@ &key (database *default-database*)) (record-sql-command sql-expression database) (let ((res (database-execute-command sql-expression database))) - (record-sql-result res database)) - (values)) + (record-sql-result res database) + res)) (defmethod execute-command ((expr %sql-expression) &key (database *default-database*)) - (execute-command (sql-output expr database) :database database) - (values)) + (execute-command (sql-output expr database) :database database)) (defmethod query ((query-expression string) &key (database *default-database*) (result-types :auto) (flatp nil) (field-names t)) diff -Nru clsql-3.7.7.ORIG/sql/utils.lisp clsql-3.7.7/sql/utils.lisp --- clsql-3.7.7.ORIG/sql/utils.lisp 2005-11-11 17:26:08.000000000 +0100 +++ clsql-3.7.7/sql/utils.lisp 2006-10-22 18:07:04.000000000 +0200 @@ -27,28 +27,41 @@ (number (float-to-sql-string num)))) +#+sbcl +(defun fixup-empty-fraction (str) + (let ((comma (position #\. str))) + (if (and comma (< comma (1- (length str))) + (char= (char str (1+ comma)) #\e)) + (concatenate 'string (subseq str 0 comma) + ".0" (subseq str (1+ comma))) + str))) + +#-sbcl +(defun fixup-empty-fraction (str) str) + (defun float-to-sql-string (num) "Convert exponent character for SQL" (let ((str (write-to-string num :readably t))) - (cond - ((find #\f str) - (substitute #\e #\f str)) - ((find #\d str) - (substitute #\e #\d str)) - ((find #\l str) - (substitute #\e #\l str)) - ((find #\s str) - (substitute #\e #\S str)) - ((find #\F str) - (substitute #\e #\F str)) - ((find #\D str) - (substitute #\e #\D str)) - ((find #\L str) - (substitute #\e #\L str)) - ((find #\S str) - (substitute #\e #\S str)) - (t - str)))) + (fixup-empty-fraction + (cond + ((find #\f str) + (substitute #\e #\f str)) + ((find #\d str) + (substitute #\e #\d str)) + ((find #\l str) + (substitute #\e #\l str)) + ((find #\s str) + (substitute #\e #\S str)) + ((find #\F str) + (substitute #\e #\F str)) + ((find #\D str) + (substitute #\e #\D str)) + ((find #\L str) + (substitute #\e #\L str)) + ((find #\S str) + (substitute #\e #\S str)) + (t + str))))) (defun sql-escape (identifier) "Change hyphens to underscores, ensure string" diff -Nru clsql-3.7.7.ORIG/tests/test-fdml.lisp clsql-3.7.7/tests/test-fdml.lisp --- clsql-3.7.7.ORIG/tests/test-fdml.lisp 2006-05-03 16:39:10.000000000 +0200 +++ clsql-3.7.7/tests/test-fdml.lisp 2006-10-21 17:55:07.000000000 +0200 @@ -634,12 +634,10 @@ ;; check status (push (clsql:in-transaction-p) results) ;; update records - (push - (clsql:with-transaction () - (clsql:update-records [employee] + (clsql:with-transaction () + (clsql:update-records [employee] :av-pairs '((email "lenin-nospam at soviet.org")) :where [= [emplid] 1])) - results) ;; check status (push (clsql:in-transaction-p) results) ;; check that was committed @@ -647,12 +645,10 @@ :flatp t) results) ;; undo the changes - (push - (clsql:with-transaction () - (clsql:update-records [employee] + (clsql:with-transaction () + (clsql:update-records [employee] :av-pairs '((email "lenin at soviet.org")) :where [= [emplid] 1])) - results) ;; and check status (push (clsql:in-transaction-p) results) ;; check that was committed @@ -660,7 +656,7 @@ :flatp t) results) (apply #'values (nreverse results))) - nil nil nil ("lenin-nospam at soviet.org") nil nil ("lenin at soviet.org")) + nil nil ("lenin-nospam at soviet.org") nil ("lenin at soviet.org")) ;; runs a valid update and an invalid one within a transaction and checks ;; that the valid update is rolled back when the invalid one fails. @@ -690,6 +686,24 @@ (apply #'values (nreverse results)))))) nil nil ("lenin at soviet.org")) +;; creates a table with test data, runs update and delete queries and checks +;; if the number of affected rows is returned correctly +(deftest :fdml/rowcount/1 + (values + (progn + (clsql:create-table [rowtest] '(([x] integer) ([y] integer))) + (loop for x upfrom 0 below 10 + sum (let ((r (clsql:insert-records + :into [rowtest] + :av-pairs `((x ,x) (y ,(1+ (random 10))))))) + (if (numberp r) r 0)))) + (clsql:update-records [rowtest] :av-pairs '((y 0)) + :where [< [x] 5]) + (prog1 + (clsql:delete-records :from [rowtest] :where [= [y] 0]) + (clsql:drop-table [rowtest] :if-does-not-exist :ignore))) + 10 5 5) + )) diff -Nru clsql-3.7.7.ORIG/tests/test-init.lisp clsql-3.7.7/tests/test-init.lisp --- clsql-3.7.7.ORIG/tests/test-init.lisp 2006-05-03 16:39:10.000000000 +0200 +++ clsql-3.7.7/tests/test-init.lisp 2006-10-23 00:07:33.173204957 +0200 @@ -627,6 +627,9 @@ '(:postgresql :mysql :sqlite3))) (clsql-sys:in test :fdml/select/37 :fdml/select/38)) (push (cons test "LIMIT keyword not supported in SELECT") skip-tests)) + ((and (eq *test-database-type* :aodbc) + (clsql-sys:in test :fdml/rowcount/1)) + (push (cons test "number of affected rows not available in aodbc") skip-tests)) (t (push test-form test-forms))))) (values (nreverse test-forms) (nreverse skip-tests)))) From stebbi at gmail.com Fri Oct 6 09:20:21 2006 From: stebbi at gmail.com (Stefan Sigurdsson) Date: Fri, 6 Oct 2006 15:20:21 +0000 Subject: [CLSQL-Help] clsql + mysql 5.0 Message-ID: <33a127360610060820s258c3b7mcc27acad01a41b5b@mail.gmail.com> Hi there, I'm new to CLSQL, and pretty excited to use it - the code samples I've looked at are very cool. I need to run against MySQL 5.0.24a but got into some configuration difficulties. In particular: SuSE Linux, MySQL 5.0.24a, SBCL 0.9.17: clsql_mysql.so needs __fixunssfdi which is local (not exported) in libmysqlclient.so. As a result clsql_mysql.so fails to load. *After* messing with this for a little while I realized that the documentation clearly states that 4.0.18 is supported (oh well :) Has anyone gotten clsql to work with mysql 5.0? And, is there a complete clsql-help list archive that I could download for grepping? Also, I used (asdf-install:install :clsql), it's minimal effort except I have to ignore db2 etc. What's the best way to get clsql installed? Is there any way to trick asdf-install into configuring only the databases I need? Hope to hear back from you guys - cheers, Stefan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.b9.com/pipermail/clsql/attachments/20061006/bfa0b6b5/attachment.htm From kevin at rosenberg.net Fri Oct 6 14:23:19 2006 From: kevin at rosenberg.net (Kevin Rosenberg) Date: Fri, 6 Oct 2006 14:23:19 -0600 Subject: [CLSQL-Help] clsql + mysql 5.0 Message-ID: <20061006202319.GA14265@rosenberg.net> On Fri, Oct 06, 2006 at 03:20:21PM +0000, Stefan Sigurdsson wrote: > SuSE Linux, MySQL 5.0.24a, SBCL 0.9.17: clsql_mysql.so needs __fixunssfdi > which is local (not exported) in libmysqlclient.so. As a result > clsql_mysql.so fails to load. I've not seen that failure. > *After* messing with this for a little while I realized that the > documentation clearly states that 4.0.18 is supported (oh well :) > Well, 4.0.18 is supported, but that's not all. http://clsql.b9.com/platforms.html is reasonably complete. > Has anyone gotten clsql to work with mysql 5.0? And, is there a complete > clsql-help list archive that I could download for grepping? I use clsql with mysql 5.0 on a frequent basis on Debian. I've also some some preliminary testing with 5.1 with good results. No, just the pipermail web interface for mail list archives. > Also, I used (asdf-install:install :clsql), it's minimal effort except I > have to ignore db2 etc. What's the best way to get clsql installed? Is there > any way to trick asdf-install into configuring only the databases I need? Download and unpack the tarball. asdf-install's method of running test on every .asd file doesn't fit well with clsql's method of using asdf systems. -- Kevin Rosenberg kevin at hypershots.com From stebbi at gmail.com Fri Oct 6 17:53:57 2006 From: stebbi at gmail.com (Stefan Sigurdsson) Date: Fri, 6 Oct 2006 23:53:57 +0000 Subject: [CLSQL-Help] clsql + mysql 5.0 In-Reply-To: <20061006200859.GA13252@b9.com> References: <33a127360610060820s258c3b7mcc27acad01a41b5b@mail.gmail.com> <20061006200859.GA13252@b9.com> Message-ID: <33a127360610061653m655cbb84u5c788c84c94e2f74@mail.gmail.com> I'll send another message if I figure out the __fixunssfdi problem on SuSE/Mysql 5.0.24a. Check out http://clsql.b9.com/manual/prerequisites.html#id2567126 , that's where I got confused about the supported platforms - one option might be to just put in a link to platform.html? I took another shot at installing, this time using a Mac with MySQL 4.0.21installed from a binary distribution that stashes things in /Library/MySQL/. I installed by exploding the tarball into sbcl/site/clsql-3.7.4, symlinking clsql.asd, clsql-uffi.asd and clsql-mysql.asd into sbcl/site-systems, and then (asdf:operate 'asdf:load-op 'clsql) (clsql:push-library-path "/Library/MySQL/lib/mysql/") (asdf:operate 'asdf:load-op 'clsql-mysql) This worked perfectly - but that slash at the end is crucial. I got warnings about failure to load libmysql.dylib, libmysqlclient.dylib, and then not finding MYSQL:MYSQL-GET-CLIENT-INFO, but all those went away when I added the slash. There, Googlable now ;) Thanks for the reply Kevin. On 10/6/06, kevinmail at boa.b9.com wrote: > > On Fri, Oct 06, 2006 at 03:20:21PM +0000, Stefan Sigurdsson wrote: > > SuSE Linux, MySQL 5.0.24a, SBCL 0.9.17: clsql_mysql.so needs > __fixunssfdi > > which is local (not exported) in libmysqlclient.so. As a result > > clsql_mysql.so fails to load. > > I've not seen that failure. > > > *After* messing with this for a little while I realized that the > > documentation clearly states that 4.0.18 is supported (oh well :) > > Well, 4.0.18 is supported, but that's not > all. http://clsql.b9.com/platforms.html is reasonably complete. > > > Has anyone gotten clsql to work with mysql 5.0? And, is there a complete > > > clsql-help list archive that I could download for grepping? > > I use clsql with mysql 5.0 on a frequent basis on Debian. I've also > some some preliminary testing with 5.1 with good results. > > No, just the pipermail web interface for mail list archives. > > > Also, I used (asdf-install:install :clsql), it's minimal effort except I > > have to ignore db2 etc. What's the best way to get clsql installed? Is > there > > any way to trick asdf-install into configuring only the databases I > need? > > Download and unpack the tarball. asdf-install's method of running test > on every .asd file doesn't fit well with clsql's method of using > asdf systems. > > Kevin > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.b9.com/pipermail/clsql/attachments/20061006/02bcdb10/attachment.htm From lnp at healy.washington.dc.us Mon Oct 9 18:37:20 2006 From: lnp at healy.washington.dc.us (Liam Healy) Date: Mon, 9 Oct 2006 20:37:20 -0400 Subject: [CLSQL-Help] Square bracket reader syntax fails Message-ID: <654935030610091737j79d049b0re4ff31b6893207d3@mail.gmail.com> I have a small set of functions using CLSQL and the [] reader syntax that I have used successfully for several years (with locally-enable-sql-reader-syntax). After an upgrade in debian on Saturday, [UPGRADE] cl-sql 3.7.2-1 -> 3.7.4-1 [UPGRADE] cl-sql-oracle 3.7.2-1 -> 3.7.4-1 [UPGRADE] cl-sql-sqlite 3.7.2-1 -> 3.7.4-1 [UPGRADE] cl-sql-sqlite3 3.7.2-1 -> 3.7.4-1 [UPGRADE] cl-sql-uffi 3.7.2-1 -> 3.7.4-1 I now find that square brackets always give "unmatched close parenthesis" in SBCL. Any ideas? (clsql:select [*] :from [lecture_summary]) READER-ERROR at 17 (line 1, column 17) on #: unmatched close parenthesis [Condition of type READER-ERROR] Restarts: 0: [ABORT-REQUEST] Abort handling SLIME request. 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (SB-IMPL::%READER-ERROR # "unmatched close parenthesis") 1: (SB-IMPL::READ-MAYBE-NOTHING # #\]) 2: (SB-IMPL::READ-LIST # #) 3: (READ-PRESERVING-WHITESPACE # NIL # T) 4: (READ-PRESERVING-WHITESPACE # NIL # NIL) 5: (READ # NIL # NIL) --more-- Liam From gilham at csl.sri.com Thu Oct 12 11:37:21 2006 From: gilham at csl.sri.com (Fred Gilham) Date: Thu, 12 Oct 2006 10:37:21 -0700 Subject: [CLSQL-Help] Reconnection logic Message-ID: <71565.1160674641@snapdragon.csl.sri.com> Hello, I'm wondering about the logic of the RECONNECT function. The problem is with connection pools. If you call RECONNECT on a database that is in a connection pool, you simply get back the same database you started with. RECONNECT calls DISCONNECT, which, in the case of a connection that is in a pool, just pushes the connection on to the connection pool's connection vector. Then RECONNECT calls CONNECT, which just pops the same connection off the connection vector of the pool. The problem comes up if you use RECONNECT for error recovery, like in the case of a database connection that has timed out. The database returned by RECONNECT will just give another error. Shouldn't RECONNECT do something to actually close a connection that is in the pool? Or is RECONNECT not useful for what I am trying to use it for? In which case how do people usually deal with the MYSQL server timing out connections? -- Fred Gilham gilham at csl.sri.com "If I thought there was anything at all in your arguments, I should have to be not only a theist, but an Episcopalian to boot," he said, after one interchange, reckoning that since Episcopalianism was, in his book, that than which nothing could be worse, this was an effective reductio ad absurdum. -- J. R. Lucas From kevin at rosenberg.net Sat Oct 14 23:53:17 2006 From: kevin at rosenberg.net (Kevin Rosenberg) Date: Sat, 14 Oct 2006 23:53:17 -0600 Subject: [CLSQL-Help] Reconnection logic In-Reply-To: <71565.1160674641@snapdragon.csl.sri.com> References: <71565.1160674641@snapdragon.csl.sri.com> Message-ID: <20061015055317.GA5264@rosenberg.net> Fred Gilham wrote: > The problem is with connection pools. If you call RECONNECT on a > database that is in a connection pool, you simply get back the same > database you started with. RECONNECT calls DISCONNECT, which, in the > case of a connection that is in a pool, just pushes the connection on to > the connection pool's connection vector. Then RECONNECT calls CONNECT, > which just pops the same connection off the connection vector of the > pool. I think your point of RECONNECT *actually* reconnecting for a pooled connect makes sense. Users of pooled connections typically just use CONNECT and DISCONNECT to get and return pooled connections to the cache. > Shouldn't RECONNECT do something to actually close a connection that is > in the pool? Yes, I think that makes sense. I'll make that change in the next release of CLSQL. > Or is RECONNECT not useful for what I am trying to use it for? In which > case how do people usually deal with the MYSQL server timing out > connections? One current way is to use DISCONNECT-POOLED to actually disconnect all pooled connections and clear the cache. -- Kevin Rosenberg kevin at hypershots.com From nimalan.mahendran at gmail.com Sun Oct 15 19:09:43 2006 From: nimalan.mahendran at gmail.com (Nimalan Mahendran) Date: Sun, 15 Oct 2006 18:09:43 -0700 Subject: [CLSQL-Help] Tables not being dropped by drop-view-from-class Message-ID: Hello, I'm using SBCL 0.9.17 with CLSQL 3.7.2 and Postgres 8.1.4 on Gentoo Linux and I'm having some problems with drop-view-from-class. create-view-from-class works fine, but drop-view-from-class doesn't actually delete the tables from the database. The classes aren't listed anymore when list-classes is executed, but the tables still exist in the database. The user associated with the database connection is the same as the owner of the tables in question. The database itself is owned by another user. Any ideas on why this is happening? What version of SBCL are people using? Maybe my version is too recent/unstable.. Thanks! -- Nimalan Mahendran From lnp at healy.washington.dc.us Tue Oct 17 19:37:31 2006 From: lnp at healy.washington.dc.us (Liam Healy) Date: Tue, 17 Oct 2006 21:37:31 -0400 Subject: [CLSQL-Help] Re: Square bracket reader syntax fails In-Reply-To: References: <654935030610091737j79d049b0re4ff31b6893207d3@mail.gmail.com> Message-ID: <654935030610171837v54e30808r9900d9b1a6f97f3b@mail.gmail.com> On 10/12/06, Marcus Pearce wrote: > > (replying privately since I'm not subscribed to clsql-help) > > On Mon, 9 Oct 2006, Liam Healy wrote: > > > [UPGRADE] cl-sql 3.7.2-1 -> 3.7.4-1 > > I contributed a patch that changed the square bracket reader in clsql > 3.7.4. > > > I now find that square brackets always give "unmatched close > > parenthesis" in SBCL. Any ideas? > > First, what version of SBCL are you using. Second, could you construct a > simple test case that illustrates the problem. > 0.9.17 Test case: your example. Oddly, I tried this on my computer at work, which is also running 0.9.17 but linux/amd64 instead of linux/x86, and your example works fine. > For example, with SBCL 0.9.17 (linux/x86) the attached file compiles fine > for me. > > Cheers, > Marcus > > From vamlists at gmail.com Wed Oct 25 00:49:50 2006 From: vamlists at gmail.com (Vamsee Kanakala) Date: Wed, 25 Oct 2006 12:19:50 +0530 Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 Message-ID: <453F090E.5000607@gmail.com> Hi all, I'm trying to run the clsql tests with Posgresql 8.1 (on SBCL 0.9.8/Ubuntu), I get the following error: Illegal :UTF-8 character starting at byte position 2. [Condition of type SB-IMPL::INVALID-UTF8-STARTER-BYTE] I see that this problem was discussed before, but I couldn't find a proper solution. Any pointers appreciated. Thanks, Vamsee. From hanche at math.ntnu.no Wed Oct 25 02:14:51 2006 From: hanche at math.ntnu.no (Harald Hanche-Olsen) Date: Wed, 25 Oct 2006 10:14:51 +0200 (CEST) Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 In-Reply-To: <453F090E.5000607@gmail.com> References: <453F090E.5000607@gmail.com> Message-ID: <20061025.101451.129298183.hanche@math.ntnu.no> + Vamsee Kanakala : | Illegal :UTF-8 character starting at byte position 2. | [Condition of type SB-IMPL::INVALID-UTF8-STARTER-BYTE] | | I see that this problem was discussed before, but I couldn't find a | proper solution. Any pointers appreciated. Bind sb-impl::*default-external-format* to :latin-1 around the load. Or just setf it, if you never deal with utf-8 encoded files. - Harald From nikodemus at random-state.net Wed Oct 25 06:56:41 2006 From: nikodemus at random-state.net (Nikodemus Siivola) Date: Wed, 25 Oct 2006 15:56:41 +0300 Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 References: <453F090E.5000607@gmail.com> <20061025.101451.129298183.hanche@math.ntnu.no> Message-ID: <87vem87dva.fsf@logxor.random-state.net> Harald Hanche-Olsen writes: > + Vamsee Kanakala : > > | Illegal :UTF-8 character starting at byte position 2. > | [Condition of type SB-IMPL::INVALID-UTF8-STARTER-BYTE] > | > | I see that this problem was discussed before, but I couldn't find a > | proper solution. Any pointers appreciated. > > Bind sb-impl::*default-external-format* to :latin-1 around the load. > Or just setf it, if you never deal with utf-8 encoded files. Better not use that, or at least don't be surprised when it breaks, as SB-IMPL is not the place where we keep supported interfaces... The right way to do this is to provide an :EXTERNAL-FORMAT :LATIN-1 argument to LOAD. Cheers, -- Nikodemus Schemer: "Buddha is small, clean, and serious." Lispnik: "Buddha is big, has hairy armpits, and laughs." From hanche at math.ntnu.no Wed Oct 25 10:02:05 2006 From: hanche at math.ntnu.no (Harald Hanche-Olsen) Date: Wed, 25 Oct 2006 18:02:05 +0200 (CEST) Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 In-Reply-To: <87vem87dva.fsf@logxor.random-state.net> References: <453F090E.5000607@gmail.com> <20061025.101451.129298183.hanche@math.ntnu.no> <87vem87dva.fsf@logxor.random-state.net> Message-ID: <20061025.180205.266845633.hanche@math.ntnu.no> + Nikodemus Siivola : | Harald Hanche-Olsen writes: | | > Bind sb-impl::*default-external-format* to :latin-1 around the load. | > Or just setf it, if you never deal with utf-8 encoded files. | | Better not use that, or at least don't be surprised when it breaks, I go for alternative #2. 8-) | as SB-IMPL is not the place where we keep supported interfaces... As the double colon amply demonstrates. Perhaps I erred in assuming that it is sufficient warning to anybody that the advice is dangerous? | The right way to do this is to provide an :EXTERNAL-FORMAT :LATIN-1 | argument to LOAD. Which means asdf level hacking. (Is there even a place in asdf to specify the coding of source files?) I agree this is the long term solution, but until maintainers of various source packages begin to take pains to specify the encoding of their files, one does what one must do. And deals with the breakage when it happens. - Harald From vamlists at gmail.com Wed Oct 25 11:57:42 2006 From: vamlists at gmail.com (Vamsee Kanakala) Date: Wed, 25 Oct 2006 23:27:42 +0530 Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 In-Reply-To: <87vem87dva.fsf@logxor.random-state.net> References: <453F090E.5000607@gmail.com> <20061025.101451.129298183.hanche@math.ntnu.no> <87vem87dva.fsf@logxor.random-state.net> Message-ID: <453FA596.70806@gmail.com> Nikodemus Siivola wrote: > Harald Hanche-Olsen writes: > >> Bind sb-impl::*default-external-format* to :latin-1 around the load. >> Or just setf it, if you never deal with utf-8 encoded files. >> > > Better not use that, or at least don't be surprised when it breaks, > as SB-IMPL is not the place where we keep supported interfaces... > > The right way to do this is to provide an :EXTERNAL-FORMAT :LATIN-1 > argument to LOAD. > > Thanks Harald & Nikodemus, I will try that out. But what I don't understand is why this is happening. Just checked, the databases I created are in fact UTF-8. In which case, why would I need to set it to latin-1? Also, I'm using slime to load up sbcl. Can anybody please point me where the "load" happens in this case? (I think it's somewhere in slime.el, but I'm not sure how to pass these params). Thanks, Vamsee. From hanche at math.ntnu.no Thu Oct 26 02:14:45 2006 From: hanche at math.ntnu.no (Harald Hanche-Olsen) Date: Thu, 26 Oct 2006 10:14:45 +0200 (CEST) Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 In-Reply-To: <453FA596.70806@gmail.com> References: <20061025.101451.129298183.hanche@math.ntnu.no> <87vem87dva.fsf@logxor.random-state.net> <453FA596.70806@gmail.com> Message-ID: <20061026.101445.74652115.hanche@math.ntnu.no> + Vamsee Kanakala : [ context snipped ] | Thanks Harald & Nikodemus, I will try that out. But what I don't | understand is why this is happening. Just checked, the databases I | created are in fact UTF-8. In which case, why would I need to set it | to latin-1? Also, I'm using slime to load up sbcl. Can anybody | please point me where the "load" happens in this case? (I think it's | somewhere in slime.el, but I'm not sure how to pass these params). Um, I think I may have read your original post too quickly. I had assumed the error happened while loading source code, but now I see no indication that this is the case. I think you need first of all to make sure that your slime connection can handle UTF-8, independently of any databases and such. Try this for example: (princ #\greek_capital_letter_omega) If it prints a capital omega, then communication between emacs and sbcl works as it should, at least in one direction. You could try pasting the resultant omega into a string and submit it as input, then run (string * 0). It should return #\GREEK_CAPITAL_LETTER_OMEGA. If all that works right, then your slime setup is almost certainly not to blame. Maybe if you provided a bit of traceback from the error, we could help further. - Harald From nikodemus at random-state.net Thu Oct 26 02:32:07 2006 From: nikodemus at random-state.net (Nikodemus Siivola) Date: Thu, 26 Oct 2006 11:32:07 +0300 Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 References: <20061025.101451.129298183.hanche@math.ntnu.no> <87vem87dva.fsf@logxor.random-state.net> <453FA596.70806@gmail.com> <20061026.101445.74652115.hanche@math.ntnu.no> Message-ID: <87d58fmq9k.fsf@logxor.random-state.net> Harald Hanche-Olsen writes: > Try this for example: (princ #\greek_capital_letter_omega) ...and add (setq slime-net-coding-system 'utf-8-unix) to your .emacs, even if it works (unless you already have it there, of course). Cheers, -- Nikodemus Schemer: "Buddha is small, clean, and serious." Lispnik: "Buddha is big, has hairy armpits, and laughs." From vamlists at gmail.com Fri Oct 27 09:18:38 2006 From: vamlists at gmail.com (Vamsee Kanakala) Date: Fri, 27 Oct 2006 20:48:38 +0530 Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 In-Reply-To: <87d58fmq9k.fsf@logxor.random-state.net> References: <20061025.101451.129298183.hanche@math.ntnu.no> <87vem87dva.fsf@logxor.random-state.net> <453FA596.70806@gmail.com> <20061026.101445.74652115.hanche@math.ntnu.no> <87d58fmq9k.fsf@logxor.random-state.net> Message-ID: <4542234E.8010003@gmail.com> Nikodemus Siivola wrote: > Harald Hanche-Olsen writes: > > >> Try this for example: (princ #\greek_capital_letter_omega) >> > > ...and add > > (setq slime-net-coding-system 'utf-8-unix) > Looks like it's not slime's fault, as I thought I should test if everything is working on a bash shell (gnome-terminal) and then try the slime part. On gnome-terminal, the omega was printed neatly without any problems. However, when I run the clsql tests like this: * (asdf:operate 'asdf:test-op 'clsql) ; loading system definition from /home/vamsee/.sbcl/systems/clsql-tests.asd ; into # ; registering # as CLSQL-TESTS ; loading system definition from /home/vamsee/.sbcl/systems/rt.asd into ; # ; registering # as RT ; loading system definition from ; /home/vamsee/.sbcl/systems/clsql-postgresql-socket.asd into #; registering # as ; CLSQL-POSTGRESQL-SOCKET ; loading system definition from ; /usr/local/lib/sbcl/sb-bsd-sockets/sb-bsd-sockets.asd into # ; registering # as SB-BSD-SOCKETS ; registering # as SB-BSD-SOCKETS-TESTS debugger invoked on a SB-IMPL::END-OF-INPUT-IN-CHARACTER in thread #: Illegal :UTF-8 character starting at byte position 3. Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. restarts (invokable by number or by possibly-abbreviated name): 0: [USE-VALUE] Supply a replacement string designator. 1: [RETRY ] Retry performing # on #. 2: [ACCEPT ] Continue, treating # on # as having been successful. 3: Retry performing # on #. 4: Continue, treating # on # as having been successful. 5: [ABORT ] Exit debugger, returning to top level. (SB-IMPL::DECODING-ERROR #(58 87 82 199) 3 4 :UTF-8 SB-IMPL::END-OF-INPUT-IN-CHARACTER 3) 0] So, would this be a problem with clsql tests or sbcl? Thanks, Vamsee. From hanche at math.ntnu.no Fri Oct 27 16:16:48 2006 From: hanche at math.ntnu.no (Harald Hanche-Olsen) Date: Sat, 28 Oct 2006 00:16:48 +0200 (CEST) Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 In-Reply-To: <4542234E.8010003@gmail.com> References: <20061026.101445.74652115.hanche@math.ntnu.no> <87d58fmq9k.fsf@logxor.random-state.net> <4542234E.8010003@gmail.com> Message-ID: <20061028.001648.74740569.hanche@math.ntnu.no> + Vamsee Kanakala : | So, would this be a problem with clsql tests or sbcl? No idea. I asked for a backtrace; you provided none. Please type the magical word "backtrace" at the debugger prompt, or use the slime debugger's backtrace (which will appear unbidden). We need to know if the stuff it chokes on comes from a file or a network socket, or whatever. A backtrace just might help with that. You are of course not prohibited from poking around in the debugger on your own to see if you can find out anything interesting. The slime debugger is particularly nice to work with, and time spent learning how to use it is time well spent. - Harald PS. FWIW, here is the vector shown in the error decoded as latin-1: cl-user> (loop for x across #(58 87 82 199) for y = (code-char x) collect y do (princ y)) :WR? (#\: #\W #\R #\LATIN_CAPITAL_LETTER_C_WITH_CEDILLA) Strange stuff to find in a file. From vamlists at gmail.com Fri Oct 27 18:31:35 2006 From: vamlists at gmail.com (Vamsee Kanakala) Date: Sat, 28 Oct 2006 06:01:35 +0530 Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 In-Reply-To: <20061028.001648.74740569.hanche@math.ntnu.no> References: <20061026.101445.74652115.hanche@math.ntnu.no> <87d58fmq9k.fsf@logxor.random-state.net> <4542234E.8010003@gmail.com> <20061028.001648.74740569.hanche@math.ntnu.no> Message-ID: <4542A4E7.7000009@gmail.com> Harald Hanche-Olsen wrote: > + Vamsee Kanakala : > > | So, would this be a problem with clsql tests or sbcl? > > No idea. I asked for a backtrace; you provided none. Please type the > magical word "backtrace" at the debugger prompt, or use the slime > debugger's backtrace (which will appear unbidden). Sorry, it's rather embarrassing, but I didn't know what exactly a backtrace was. Here goes: 0: (SB-IMPL::DECODING-ERROR #(32 175 188 6) 1 2 :UTF-8 SB-IMPL::INVALID-UTF8-STARTER-BYTE 1) 1: (SB-IMPL::BYTES-PER-UTF8-CHARACTER-AREF # # #) 2: (SB-IMPL::UTF8->STRING-AREF #(32 175 188 6) 0 4) 3: (POSTGRESQL-SOCKET:REOPEN-POSTGRESQL-CONNECTION #S(POSTGRESQL-SOCKET:POSTGRESQL-CONNECTION :HOST "localhost" :PORT 5432 :DATABASE "clsql-test" :USER "vamsee" :PASSWORD "xxxxxx" :OPTIONS "" :TTY "" :SOCKET # :PID NIL :KEY NIL)) 4: ((SB-PCL::FAST-METHOD CLSQL-SYS:DATABASE-CONNECT (T (EQL :POSTGRESQL-SOCKET))) # # ("localhost" "clsql-test" "vamsee" "xxxxxx") :POSTGRESQL-SOCKET) 5: (CLSQL-SYS:CONNECT ("localhost" "clsql-test" "vamsee" "xxxxxx") :IF-EXISTS :OLD :MAKE-DEFAULT T :POOL NIL :DATABASE-TYPE :POSTGRESQL-SOCKET) 6: (CLSQL-TESTS:TEST-CONNECT-TO-DATABASE :POSTGRESQL-SOCKET ("localhost" "clsql-test" "vamsee" "xxxxxx")) 7: (CLSQL-TESTS::DO-TESTS-FOR-BACKEND :POSTGRESQL-SOCKET ("localhost" "clsql-test" "vamsee" "xxxxxx")) 8: (CLSQL-TESTS:RUN-TESTS :REPORT-STREAM NIL :SEXP-REPORT-STREAM NIL) 9: ((SB-PCL::FAST-METHOD ASDF:PERFORM (ASDF:TEST-OP (EQL (ASDF:FIND-SYSTEM 'CLSQL-TESTS-SYSTEM::CLSQL-TESTS)))) # # # #) 10: ((LAMBDA (SB-PCL::.PV-CELL. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. SB-PCL::.ARG1.)) # # # #) 11: ((LAMBDA ())) 12: (SB-C::%WITH-COMPILATION-UNIT #) 13: (ASDF:OPERATE ASDF:TEST-OP CLSQL-SYSTEM::CLSQL-TESTS :FORCE T) 14: ((LAMBDA (SB-PCL::.PV-CELL. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. SB-PCL::.ARG1.)) # # # #) 15: ((LAMBDA ())) 16: (SB-C::%WITH-COMPILATION-UNIT #) 17: (ASDF:OPERATE ASDF:TEST-OP CLSQL) 18: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ASDF:OPERATE 'ASDF:TEST-OP 'CLSQL) #) 19: (INTERACTIVE-EVAL (ASDF:OPERATE 'ASDF:TEST-OP 'CLSQL)) 20: (SB-IMPL::REPL-FUN NIL) 21: ((LAMBDA ())) 22: ((LAMBDA ())) 23: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #) 24: (SB-IMPL::TOPLEVEL-REPL NIL) 25: (SB-IMPL::TOPLEVEL-INIT) 26: ((LABELS SB-IMPL::RESTART-LISP)) Thanks, Vamsee. From jcm at FreeBSD-uk.eu.org Fri Oct 27 22:12:15 2006 From: jcm at FreeBSD-uk.eu.org (Jonathon McKitrick) Date: Sat, 28 Oct 2006 05:12:15 +0100 Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 In-Reply-To: <4542A4E7.7000009@gmail.com> References: <20061026.101445.74652115.hanche@math.ntnu.no> <87d58fmq9k.fsf@logxor.random-state.net> <4542234E.8010003@gmail.com> <20061028.001648.74740569.hanche@math.ntnu.no> <4542A4E7.7000009@gmail.com> Message-ID: <20061028041215.GA2789@dogma.freebsd-uk.eu.org> On Sat, Oct 28, 2006 at 06:01:35AM +0530, Vamsee Kanakala wrote: : 5: (CLSQL-SYS:CONNECT : ("localhost" "clsql-test" "vamsee" "xxxxxx") : :IF-EXISTS : :OLD : :MAKE-DEFAULT : T : :POOL : NIL : :DATABASE-TYPE : :POSTGRESQL-SOCKET) I've seen something similar happen as well. I just decided to run postgresql-socket on macs and regular postgresql (with ffi) on Linux. Jonathon McKitrick -- My other computer is your Windows box. From hanche at math.ntnu.no Sat Oct 28 04:17:19 2006 From: hanche at math.ntnu.no (Harald Hanche-Olsen) Date: Sat, 28 Oct 2006 12:17:19 +0200 (CEST) Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 In-Reply-To: <4542A4E7.7000009@gmail.com> References: <4542234E.8010003@gmail.com> <20061028.001648.74740569.hanche@math.ntnu.no> <4542A4E7.7000009@gmail.com> Message-ID: <20061028.121719.74702090.hanche@math.ntnu.no> + Vamsee Kanakala : | 0: (SB-IMPL::DECODING-ERROR | #(32 175 188 6) | [...] | 2: (SB-IMPL::UTF8->STRING-AREF #(32 175 188 6) 0 4) | 3: (POSTGRESQL-SOCKET:REOPEN-POSTGRESQL-CONNECTION Thank you. As far as I can get from the source code for reopen-postgresql-connection, this must mean that your postgresql server (aka postmaster) requested your password as an MD5 hash. It has just sent you the four octets #(32 175 188 6) as the salt to be used for the password hash (this would explain why the value varies), but code in reopen-postgresql-connection tries to decode this as a utf-8 string (via (read-socket-sequence socket 4)) which fails, of course. It is interesting to note that read-socket-sequence assumes the input is a unicode string if sbcl has been compiled with unicode support. Making assumptions about network protocols based merely on properties of the Lisp implementation seems totally inappropriate to me, so this is clearly a clsql bug, in my opinion. As an experiment, you might try to configure your postgresql server to use plaintext passwords instead of md5 hashes. If you never intend to access the database over the network, the security implications are minimal. And if this cures the problem, the code in postgresql-socket-api.lisp should be relatively easy to fix: A variant of read-socket-sequence (call it read-socket-byte-sequence perhaps) is needed that is just like the #-sb-unicode branch of read-socket-sequence, and that can be called from the appropriate places within reopen-postgresql-connection. But OTOH, quite possible there are more places where the encoding gets mixed up in this way, in which case I am not the right person to suggest a fix. - Harald From jcm at FreeBSD-uk.eu.org Sat Oct 28 08:57:52 2006 From: jcm at FreeBSD-uk.eu.org (Jonathon McKitrick) Date: Sat, 28 Oct 2006 15:57:52 +0100 Subject: [CLSQL-Help] Bracket notation an advantage? Message-ID: <20061028145751.GA5311@dogma.freebsd-uk.eu.org> I'm trying to decide if bracket notation is gaining me anything versus straight sql strings. Let's assume for now I do not want an OO approach, since I prefer lists for a variety of reasons. What do I gain from (clsql:select [*] :from [customers] :order-by [cust_id]) versus something like (exec-sql "select * from customers order by cust_id") after writing exec-sql, of course? Jonathon McKitrick -- My other computer is your Windows box. From hanche at math.ntnu.no Sat Oct 28 09:19:09 2006 From: hanche at math.ntnu.no (Harald Hanche-Olsen) Date: Sat, 28 Oct 2006 17:19:09 +0200 (CEST) Subject: [CLSQL-Help] Bracket notation an advantage? In-Reply-To: <20061028145751.GA5311@dogma.freebsd-uk.eu.org> References: <20061028145751.GA5311@dogma.freebsd-uk.eu.org> Message-ID: <20061028.171909.246503761.hanche@math.ntnu.no> + Jonathon McKitrick : | What do I gain from | (clsql:select [*] :from [customers] :order-by [cust_id]) | | versus something like | | (exec-sql "select * from customers order by cust_id") Nothing much. But presumably, you gain something from being able to write (clsql:select [*] :from [customers] :where [= [cust_id] customer-id]) rather than having to build the query using (format nil "select * from customers where custid='~A'" customer-id) Building query strings with format is doable, but quickly becomes unwieldy. Plus, the former solution does doubles any single quotes within customer-id appropriately: (clsql:sql [= [cust_id] "John's Auto"]) -> "(CUST_ID = 'John''s Auto')" - Harald From vamlists at gmail.com Sat Oct 28 10:59:19 2006 From: vamlists at gmail.com (Vamsee Kanakala) Date: Sat, 28 Oct 2006 22:29:19 +0530 Subject: [CLSQL-Help] UTF-8 problem with postgresql 8.1 In-Reply-To: <20061028.121719.74702090.hanche@math.ntnu.no> References: <4542234E.8010003@gmail.com> <20061028.001648.74740569.hanche@math.ntnu.no> <4542A4E7.7000009@gmail.com> <20061028.121719.74702090.hanche@math.ntnu.no> Message-ID: <45438C67.4080904@gmail.com> Harald Hanche-Olsen wrote: [snipped] > As an experiment, you might try to configure your postgresql server to > use plaintext passwords instead of md5 hashes. That was very helpful analysis. As the webapp will be living only on my dev machine for sometime, I set the localhost connections to 'trust' and sent an empty string as password in the .clsql-test.config file. The tests are passing now. Many thanks. Vamsee.