All Projects → dingmaotu → mql-sqlite3

dingmaotu / mql-sqlite3

Licence: MIT License
SQLite3 binding for the MQL language (both 32bit MT4 and 64bit MT5)

Programming Languages

MQL5
34 projects
MQL4
33 projects

mql-sqlite3

SQLite3 binding for the MQL language (both 32bit MT4 and 64bit MT5)

Introduction

This is a complete binding of the SQLite3 library (version 3180000) for the MQL4/5 language used by MetaTrader4/5.

SQLite is an fast embeded SQL engine written in C and widely used by a lot of projects. We always have the need to persistent states of Expert Advisors and SQLite seems to be the best solution.

This binding tries to remain compatible between MQL4/5. Users of both versions can use this binding, with a single set of headers. MQL4 and MQL5 are basically the same in that they are merged in recent versions. The difference is in the runtime environment (MetaTrader5 is 64bit by default, while MetaTrader4 is 32bit). The trading system is also different, but it is no concern of this binding.

Files

This binding contains three sets of files:

  1. The binding itself is in the Include/SQLite3 directory.

  2. There is a simple testing script called TestSQLite3.mq4 in Scripts/Test directory. The script files are mq4 by default, but you can change the extension to mq5 to use them in MetaTrader5. Currently there is only one script. I am going to add more in the future.

  3. Precompiled DLLs of both 64bit (Library/MT5) and 32bit (Library/MT4) are provided. Copy the corresponding DLLs to the Library folder of your MetaTrader terminal. If you are using MetaTrader5 32bit, use the 32bit version from Library/MT4. The DLLs have no special dependencies and should work in any Windows version after NT. Note that these DLLs are copied from official binary release, without any modification. You can download/compile your own if you don't trust these binaries.

API comparison

Below is a detailed comparison table of MQL and C/C++ APIs. If you are reading the official documentation, and you want to find the corresponding MQL API for the C API, use this table. This table also has comments about why a particular API is not included in this binding. This table might change if new APIs get added.

SQLite3 C API mql-sqlite3 Comment
sqlite3_aggregate_context N/A for extension
sqlite3_aggregate_count N/A deprecated
sqlite3_auto_extension N/A for extension
sqlite3_backup_finish Backup::~Backup  
sqlite3_backup_init Backup::Backup  
sqlite3_backup_pagecount Backup::getPageCount  
sqlite3_backup_remaining Backup::getRemaining  
sqlite3_backup_step Backup::step  
sqlite3_bind_blob Statement::bind  
sqlite3_bind_blob64 N/A too large for ordinary use
sqlite3_bind_double Statement::bind  
sqlite3_bind_int Statement::bind  
sqlite3_bind_int64 Statement::bind  
sqlite3_bind_null Statement::bind  
sqlite3_bind_parameter_count Statement::getParameterCount  
sqlite3_bind_parameter_index Statement::getParameterIndex  
sqlite3_bind_parameter_name Statement::getParameterName  
sqlite3_bind_text Statement::bind  
sqlite3_bind_text16 N/A always UTF-8
sqlite3_bind_text64 N/A too large for ordinary use
sqlite3_bind_value N/A for extension
sqlite3_bind_zeroblob Statement::zero  
sqlite3_bind_zeroblob64 Statement::zero  
sqlite3_blob_bytes Blob::size  
sqlite3_blob_close Blob::~Blob  
sqlite3_blob_open Blob::Blob  
sqlite3_blob_read Blob::read  
sqlite3_blob_reopen Blob::moveTo  
sqlite3_blob_write Blob::write  
sqlite3_busy_handler N/A Need callback
sqlite3_busy_timeout SQLite3::setBusyTimeout  
sqlite3_cancel_auto_extension N/A for extension
sqlite3_changes SQLite3::getChanges  
sqlite3_clear_bindings Statement::clearBindings  
sqlite3_close N/A use v2
sqlite3_close_v2 SQLite3::~SQLite3  
sqlite3_collation_needed N/A for extension
sqlite3_collation_needed16 N/A for extension
sqlite3_column_blob Statement::getColumn  
sqlite3_column_bytes Statement::getColumnBytes  
sqlite3_column_bytes16 N/A always UTF-8
sqlite3_column_count Statement::getColumnCount  
sqlite3_column_database_name Statement::getColumnDatabaseName  
sqlite3_column_database_name16 N/A always UTF-8
sqlite3_column_decltype Statement::getColumnDeclareType  
sqlite3_column_decltype16 N/A always UTF-8
sqlite3_column_double Statement::getColumn  
sqlite3_column_int Statement::getColumn  
sqlite3_column_int64 Statement::getColumn  
sqlite3_column_name Statement::getColumnName  
sqlite3_column_name16 N/A always UTF-8
sqlite3_column_origin_name Statement::getColumnOriginName  
sqlite3_column_origin_name16 N/A always UTF-8
sqlite3_column_table_name Statement::getColumnTableName  
sqlite3_column_table_name16 N/A always UTF-8
sqlite3_column_text Statement::getColumn  
sqlite3_column_text16 N/A always UTF-8
sqlite3_column_type Statement::getColumnType  
sqlite3_column_value N/A for extension
sqlite3_commit_hook N/A need callback
sqlite3_compileoption_get N/A diagnostic/optional
sqlite3_compileoption_used N/A diagnostic/optional
sqlite3_complete Statement::isComplete  
sqlite3_complete16 N/A always UTF-8
sqlite3_config SQLite3::set* static methods partial
sqlite3_context_db_handle N/A for extension
sqlite3_create_collation N/A for extension
sqlite3_create_collation16 N/A for extension
sqlite3_create_collation_v2 N/A for extension
sqlite3_create_function N/A for extension
sqlite3_create_function16 N/A for extension
sqlite3_create_function_v2 N/A for extension
sqlite3_create_module N/A for extension
sqlite3_create_module_v2 N/A for extension
sqlite3_data_count Statement::getDataCount  
sqlite3_db_cacheflush SQLite3::flush  
sqlite3_db_config SQLite3::setDbLookAside
SQLite3::setMainDbName
SQLite3::isTriggerEnabled
SQLite3::setTriggerEnabled
SQLite3::isForeignKeyEnabled
SQLite3::setForeignKeyEnabled
SQLite3::isFTS3TokenizerEnabled
SQLite3::setFTS3TokenizerEnabled
SQLite3::isLoadExtensionEnabled
SQLite3::setLoadExtensionEnabled
SQLite3::isCheckpointsOnCloseEnabled
SQLite3::setCheckpointsOnCloseEnabled
 
sqlite3_db_filename SQLite3::getDbFilename  
sqlite3_db_handle Statement::getConnectionHandle  
sqlite3_db_mutex N/A internal use only
sqlite3_db_readonly SQLite3::isReadonly SQLite3::hasDb check if contains db
sqlite3_db_release_memory SQLite3::releaseMemory  
sqlite3_db_status SQLite3::getStatus  
sqlite3_declare_vtab N/A for extension
sqlite3_enable_load_extension SQLite3::setLoadExtension  
sqlite3_enable_shared_cache SQLite3::setSharedCache  
sqlite3_errcode SQLite3::getErrorCode  
sqlite3_errmsg SQLite3::getErrorMsg  
sqlite3_errmsg16 N/A always UTF-8
sqlite3_errstr SQLite3::errorCode2Msg  
sqlite3_exec N/A convenience wrapper only
sqlite3_expanded_sql SQLite3::getExpandedSql  
sqlite3_expired N/A deprecated
sqlite3_extended_errcode SQLite3::getExtendedErrorCode  
sqlite3_extended_result_codes SQLite3::setResultCodes  
sqlite3_file_control N/A too low level
sqlite3_finalize SQLite3::~SQLite3  
sqlite3_free N/A internal use only
sqlite3_free_table N/A convenience wrapper only
sqlite3_get_autocommit SQLite3::isAutoCommit  
sqlite3_get_auxdata N/A for extension
sqlite3_get_table N/A convenience wrapper only
sqlite3_global_recover N/A deprecated
sqlite3_initialize SQLite3::initialize static
sqlite3_interrupt SQLite3::interrupt  
sqlite3_last_insert_rowid SQLite3::getLastInsertRowId  
sqlite3_libversion SQLite3::getVersion static
sqlite3_libversion_number SQLite3::getVersionNumber static
sqlite3_limit SQLite3::setLimit  
sqlite3_load_extension SQLite3::loadExtension  
sqlite3_log N/A for extension
sqlite3_malloc N/A internal use only
sqlite3_malloc64 N/A internal use only
sqlite3_memory_alarm N/A deprecated
sqlite3_memory_highwater SQLite3::getMemoryHighwater, SQLite3::resetMemoryHighwater
sqlite3_memory_used SQLite3::getMemoryUsed  
sqlite3_mprintf N/A internal use only
sqlite3_msize N/A internal use only
sqlite3_mutex_alloc N/A internal use only
sqlite3_mutex_enter N/A internal use only
sqlite3_mutex_free N/A internal use only
sqlite3_mutex_held N/A internal use only
sqlite3_mutex_leave N/A internal use only
sqlite3_mutex_notheld N/A internal use only
sqlite3_mutex_try N/A internal use only
sqlite3_next_stmt N/A no use
sqlite3_open N/A use v2
sqlite3_open16 N/A always UTF-8
sqlite3_open_v2 SQLite3::SQLite3  
sqlite3_os_end N/A internal use only
sqlite3_os_init N/A internal use only
sqlite3_overload_function N/A for extension
sqlite3_prepare N/A use v2
sqlite3_prepare16 N/A always UTF-8
sqlite3_prepare16_v2 N/A always UTF-8
sqlite3_prepare_v2 Statement::Statement  
sqlite3_preupdate_count N/A for extension
sqlite3_preupdate_depth N/A for extension
sqlite3_preupdate_hook N/A for extension
sqlite3_preupdate_new N/A for extension
sqlite3_preupdate_old N/A for extension
sqlite3_profile N/A deprecated
sqlite3_progress_handler N/A need callback
sqlite3_randomness N/A use MQL Random
sqlite3_realloc N/A internal use only
sqlite3_realloc64 N/A internal use only
sqlite3_release_memory SQLite3::releaseMemoryInBytes static
sqlite3_reset Statement::reset  
sqlite3_reset_auto_extension N/A for extension
sqlite3_result_blob N/A for extension
sqlite3_result_blob64 N/A for extension
sqlite3_result_double N/A for extension
sqlite3_result_error N/A for extension
sqlite3_result_error16 N/A for extension
sqlite3_result_error_code N/A for extension
sqlite3_result_error_nomem N/A for extension
sqlite3_result_error_toobig N/A for extension
sqlite3_result_int N/A for extension
sqlite3_result_int64 N/A for extension
sqlite3_result_null N/A for extension
sqlite3_result_subtype N/A for extension
sqlite3_result_text N/A for extension
sqlite3_result_text16 N/A for extension
sqlite3_result_text16be N/A for extension
sqlite3_result_text16le N/A for extension
sqlite3_result_text64 N/A for extension
sqlite3_result_value N/A for extension
sqlite3_result_zeroblob N/A for extension
sqlite3_result_zeroblob64 N/A for extension
sqlite3_rollback_hook N/A need callback
sqlite3_set_authorizer N/A need callback
sqlite3_set_auxdata N/A for extension
sqlite3_set_last_insert_rowid SQLite3::setLastInsertRowId  
sqlite3_shutdown SQLite3::shutdown  
sqlite3_sleep N/A use MQL Sleep
sqlite3_snapshot_cmp(exp) N/A experimental
sqlite3_snapshot_free(exp) N/A experimental
sqlite3_snapshot_get(exp) N/A experimental
sqlite3_snapshot_open(exp) N/A experimental
sqlite3_snapshot_recover(exp) N/A experimental
sqlite3_snprintf N/A internal use only
sqlite3_soft_heap_limit N/A deprecated
sqlite3_soft_heap_limit64 SQLite3::setSoftHeapLimit static
sqlite3_sourceid SQLite3::getSourceId static
sqlite3_sql Statement::getSql  
sqlite3_status N/A use sqlite3_status64
sqlite3_status64 SQLite3::status  
sqlite3_step Statement::step  
sqlite3_stmt_busy Statement::isBusy  
sqlite3_stmt_readonly Statement::isReadonly  
sqlite3_stmt_scanstatus N/A advanced/optional
sqlite3_stmt_scanstatus_reset N/A advanced/optional
sqlite3_stmt_status Statement::getCounter, Statement::resetCounter  
sqlite3_strglob N/A internal use only
sqlite3_stricmp N/A internal use only
sqlite3_strlike N/A internal use only
sqlite3_strnicmp N/A internal use only
sqlite3_system_errno N/A internal use only
sqlite3_table_column_metadata SQLite3::getDbColumnMetadata  
sqlite3_test_control N/A internal use only
sqlite3_thread_cleanup N/A deprecated
sqlite3_threadsafe SQLite3::isThreadSafe  
sqlite3_total_changes SQLite3::getTotalChanges  
sqlite3_trace N/A deprecated
sqlite3_trace_v2 N/A need callback
sqlite3_transfer_bindings N/A deprecated
sqlite3_unlock_notify N/A need callback
sqlite3_update_hook N/A need callback
sqlite3_uri_boolean N/A for extension
sqlite3_uri_int64 N/A for extension
sqlite3_uri_parameter N/A for extension
sqlite3_user_data N/A for extension
sqlite3_value_blob N/A for extension
sqlite3_value_bytes N/A for extension
sqlite3_value_bytes16 N/A for extension
sqlite3_value_double N/A for extension
sqlite3_value_dup N/A for extension
sqlite3_value_free N/A for extension
sqlite3_value_int N/A for extension
sqlite3_value_int64 N/A for extension
sqlite3_value_numeric_type N/A for extension
sqlite3_value_subtype N/A for extension
sqlite3_value_text N/A for extension
sqlite3_value_text16 N/A for extension
sqlite3_value_text16be N/A for extension
sqlite3_value_text16le N/A for extension
sqlite3_value_type N/A for extension
sqlite3_version N/A not a function
sqlite3_vfs_find Vfs::find, Vfs::getDefault  
sqlite3_vfs_register Vfs::register, Vfs::registerDefault  
sqlite3_vfs_unregister Vfs::unregister  
sqlite3_vmprintf N/A internal use only
sqlite3_vsnprintf N/A internal use only
sqlite3_vtab_config N/A for extension
sqlite3_vtab_on_conflict N/A for extension
sqlite3_wal_autocheckpoint SQLite3::setAutoCheckpoint  
sqlite3_wal_checkpoint N/A use v2
sqlite3_wal_checkpoint_v2 SQLite3::checkpoint  
sqlite3_wal_hook N/A need callback

About string encoding and API choices

MQL strings are Win32 UNICODE strings (basically 2-byte UTF-16). While for several APIs, SQLite3 provides both UTF-8 and UTF-16 versions, only UTF-8 strings are accepted in other APIs. For unification and maximum interoperabibility, we only use the UTF-8 version of the APIs. In this binding all strings are converted to utf-8 strings before sending to the dll layer. And strings in the database are also UTF-8 encoded.

About callbacks, internal APIs, and convenience wrappers

A lot of SQLite3 APIs are for internal use only, or for extension development, or need a C function as callback. These are not included in this binding. This will not affect the general applicability of this binding for most application level usage.

APIs like sqlite3_exec, sqlite3_get_table, etc. are convenience wrappers, which are not included. Maybe I will implement them in MQL.

Usage

You'd better read the official documentation or some books about SQLite before using this library. Documentation about this binding will be added when I have the time to finish them. You can find a simple test script in Scripts/Test. Here is a sample from TestSQLite3.mq4:

//+------------------------------------------------------------------+
//|                                                  TestSQLite3.mq4 |
//|                                          Copyright 2017, Li Ding |
//|                                            [email protected] |
//+------------------------------------------------------------------+
#property copyright "Copyright 2017, Li Ding"
#property link      "[email protected]"
#property version   "1.00"
#property strict

#include <SQLite3/Statement.mqh>
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
//--- optional but recommended
   SQLite3::initialize();

//--- ensure the dll and the lib is of the same version
   Print(SQLite3::getVersionNumber(), " = ", SQLITE_VERSION_NUMBER);
   Print(SQLite3::getVersion(), " = ", SQLITE_VERSION);
   Print(SQLite3::getSourceId(), " = ", SQLITE_SOURCE_ID);

//--- create an empty db
#ifdef __MQL5__
   string filesPath=TerminalInfoString(TERMINAL_DATA_PATH)+"\\MQL5\\Files";
#else
   string filesPath=TerminalInfoString(TERMINAL_DATA_PATH)+"\\MQL4\\Files";
#endif
   string dbPath=filesPath+"\\test.db";
   Print(dbPath);

   SQLite3 db(dbPath,SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE);
   if(!db.isValid()) return;

   Print("DB created.");
   string sql="create table buy_orders"
              "(a int, b text);";
   if(Statement::isComplete(sql))
      Print(">>> SQL is complete");
   else
      Print(">>> SQL not complete");

   Statement s(db,sql);

   if(!s.isValid())
     {
      Print(db.getErrorMsg());
      return;
     }

   int r=s.step();
   if(r == SQLITE_OK)
      Print(">>> Step finished.");
   else if(r==SQLITE_DONE)
      Print(">>> Successfully created table.");
   else
      Print(">>> Error executing statement: ",db.getErrorMsg());

//--- optional but recommended
   SQLite3::shutdown();
  }

Here is a sample for select:

//+------------------------------------------------------------------+
//|                                            TestSQLite3Select.mq4 |
//|                                          Copyright 2018, Li Ding |
//|                                            [email protected] |
//+------------------------------------------------------------------+
#property copyright "Copyright 2018, Li Ding"
#property link      "[email protected]"
#property version   "1.00"
#property strict

#include <SQLite3/Statement.mqh>
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
//--- optional but recommended
   SQLite3::initialize();

//--- open database
#ifdef __MQL5__
   string filesPath=TerminalInfoString(TERMINAL_DATA_PATH)+"\\MQL5\\Files";
#else
   string filesPath=TerminalInfoString(TERMINAL_DATA_PATH)+"\\MQL4\\Files";
#endif
   string dbPath=filesPath+"\\test.db";
   Print(dbPath);

   SQLite3 db(dbPath,SQLITE_OPEN_READWRITE);
   if(!db.isValid()) return;

   string sql="select a, b from buy_orders;";

   Statement s(db,sql);

   if(!s.isValid())
     {
      Print(db.getErrorMsg());
      return;
     }

   int r=s.step();
   do
     {
      if(r==SQLITE_ROW)
        {
         Print(">>> New row!");
         int c=s.getColumnCount();
         for(int i=0; i<c; i++)
           {
            if(i==0)
              {
               int value;
               s.getColumn(i,value);
               Print(s.getColumnName(i),": ",value);
              }
            else if(i==1)
              {
               string value;
               s.getColumn(i,value);
               Print(s.getColumnName(i),": ",value);
              }
           }
        }
      else
         break;

      r=s.step();
     }
   while(r!=SQLITE_DONE);

//--- optional but recommended
   SQLite3::shutdown();
  }
//+------------------------------------------------------------------+

TODO

  1. Write more tests.
  2. Documentation.

Changes

  • 2017-06-21: Initial version. Alpha quality, needs more tests.
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].