GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — simplify dist by merging sql-asm-memory-growth.js into sql-asm.js by kaizhu256 · Pull Request #431 · sql-js/sql.js

Conversation

@kaizhu256

@kaizhu256 kaizhu256 commented Oct 21, 2020

Copy link
Copy Markdown
Member

Co-authored-by: Yan Li [email protected]
Co-authored-by: kai zhu [email protected]

this implements suggestion #239 (comment)

removed or edited all references to memory-growth using command git grep -i 'memory.*growth'

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

brody2consult and Taytay reacted with hooray emoji brody2consult reacted with eyes emoji
All reactions
  • 🎉 2 reactions
  • 👀 1 reaction
@kaizhu256 @yanli0303
merge sql-asm-memory-growth.js into sql-asm.js
ef1054e
Co-authored-by: Yan Li <[email protected]>
Co-authored-by: kai zhu <[email protected]>
@lovasoa

lovasoa commented Oct 21, 2020

Copy link
Copy Markdown
Member

What is the performance impact ?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@kaizhu256

kaizhu256 commented Oct 22, 2020 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

good question. the following benchmark indicates asm2 is 1-6% slower running queries on a 60mb database:

// asm2 = LEGACY_VM_SUPPORT + ALLOW_MEMORY_GROWTH
// used build-artifacts from https://github.com/sql-js/sql.js/actions/runs/320372611 for asm2 binaries
// used build-artifacts from https://github.com/sql-js/sql.js/actions/runs/316181408 for asm, wasm binaries

loading 60mb database in chrome
sampled 10 times
asm2:   20.6 +/- 2.2 ms   ( 5% slower than asm)
asm :   19.7 +/- 1.9 ms
wasm:   15.4 +/- 0.3 ms   (22% faster than asm)

copy all tables in 60mb database in chrome
sampled 5 times
asm2:   15798 +/-  47 ms  (  1% slower than asm)
asm :   15610 +/- 152 ms
wasm:    3384 +/-  47 ms  (360% faster than asm)

fts-query with joins in chrome
sampled 5 times
asm2:   40464 +/- 115 ms  (  6% slower than asm)
asm :   38171 +/- 220 ms
wasm:   13942 +/- 216 ms  (170% faster than asm)

you can emulate above result in following steps:

image

  1. download 60mb enron-email database from https://github.com/kaizhu256/sqlite-enron

  2. load database into modified sql.js demos below:

    • https://kaizhu256.github.io/sql.js/examples/GUI.asm2/
    • https://kaizhu256.github.io/sql.js/examples/GUI.asm/
    • https://kaizhu256.github.io/sql.js/examples/GUI.wasm/
  3. run following sql-queries used in benchmarks:

copy all tables in 60mb database

-- create database-schema
CREATE TABLE `employeelist2` (
  `eid` integer  NOT NULL PRIMARY KEY AUTOINCREMENT
,  `firstName` varchar(31) NOT NULL DEFAULT ''
,  `lastName` varchar(31) NOT NULL DEFAULT ''
,  `Email_id` varchar(31) NOT NULL DEFAULT ''
,  `Email2` varchar(31) DEFAULT NULL
,  `Email3` varchar(31) DEFAULT NULL
,  `EMail4` varchar(31) DEFAULT NULL
,  `folder` varchar(31) NOT NULL DEFAULT ''
,  `status` varchar(50) DEFAULT NULL
,  UNIQUE (`Email_id`)
);
CREATE TABLE `message2` (
  `mid` integer NOT NULL DEFAULT '0'
,  `sender` varchar(127) NOT NULL DEFAULT ''
,  `date` datetime DEFAULT NULL
,  `message_id` varchar(127) DEFAULT NULL
-- ,  `subject` text
-- ,  `body` text
,  `folder` varchar(127) NOT NULL DEFAULT ''
,  PRIMARY KEY (`mid`)
);
CREATE TABLE `recipientinfo2` (
  `rid` integer NOT NULL DEFAULT '0'
,  `mid` integer  NOT NULL DEFAULT '0'
,  `rtype` text  DEFAULT NULL
,  `rvalue` varchar(127) DEFAULT NULL
,  `dater` datetime DEFAULT NULL
,  PRIMARY KEY (`rid`)
,  FOREIGN KEY (mid) REFERENCES message2(mid)
);
CREATE TABLE `referenceinfo2` (
  `rfid` integer NOT NULL DEFAULT '0'
,  `mid` integer NOT NULL DEFAULT '0'
,  `reference` text
,  PRIMARY KEY (`rfid`)
,  FOREIGN KEY (mid) REFERENCES message2(mid)
);
CREATE INDEX "idx_message_sender2" ON "message2" (`sender`);
CREATE VIRTUAL TABLE messages_ft2 USING fts4(subject, body);

-- insert tables
-- employeelist2
-- message2
-- recipientinfo2
-- referenceinfo2
-- message_ft
SELECT 'employeelist', COUNT(*) FROM employeelist;
SELECT 'message', COUNT(*) FROM message;
SELECT 'recipientinfo', COUNT(*) FROM recipientinfo;
SELECT 'referenceinfo', COUNT(*) FROM referenceinfo;
SELECT 'messages_ft', COUNT(*) FROM messages_ft;

INSERT INTO message2 SELECT * FROM
    (
    SELECT
        message.mid,
        message.sender,
        message.date,
        message.message_id,
        -- message.subject,
        -- message.body,
        message.folder
    FROM
    (
    SELECT DISTINCT message.mid
    FROM message
    INNER JOIN recipientinfo ON recipientinfo.mid = message.mid
    INNER JOIN referenceinfo ON referenceinfo.mid = message.mid
    ) AS tmp1
    INNER JOIN message ON message.mid = tmp1.mid
    ORDER BY tmp1.mid
    LIMIT 10000
    ) AS tmp1;

INSERT INTO employeelist2 SELECT * FROM employeelist;
INSERT INTO recipientinfo2 SELECT
        tmp1.rid,
        tmp1.mid,
        tmp1.rtype,
        tmp1.rvalue,
        tmp1.dater
    FROM message2
    LEFT JOIN recipientinfo AS tmp1 ON tmp1.mid = message2.mid
    ORDER BY tmp1.rid;
INSERT INTO referenceinfo2 SELECT
        tmp1.rfid,
        tmp1.mid,
        tmp1.reference
    FROM message2
    LEFT JOIN referenceinfo AS tmp1 ON tmp1.mid = message2.mid
    ORDER BY tmp1.rfid;
INSERT INTO messages_ft2 (
    rowid, subject, body
)
SELECT
        tmp1.rowid, tmp1.subject, tmp1.body
    FROM message2
    INNER JOIN messages_ft AS tmp1 ON tmp1.rowid = message2.mid
    ORDER BY tmp1.rowid;
VACUUM;
SELECT 'employeelist2', COUNT(*) FROM employeelist2;
SELECT 'message2', COUNT(*) FROM message2;
SELECT 'recipientinfo2', COUNT(*) FROM recipientinfo2;
SELECT 'referenceinfo2', COUNT(*) FROM referenceinfo2;
SELECT 'messages_ft2', COUNT(*) FROM messages_ft2;
--*/

fts-query with joins

SELECT * FROM messages_ft2
INNER JOIN recipientinfo2 ON recipientinfo2.mid = messages_ft2.rowid
INNER JOIN referenceinfo2 ON referenceinfo2.mid = messages_ft2.rowid
WHERE body MATCH 'energy NEAR oil';
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@kaizhu256

kaizhu256 commented Oct 22, 2020

Copy link
Copy Markdown
Member Author

also the difference in size is < 1%:

                                 asm        asm2
sql-asm-debug.js          10,865,349  10,896,696  bytes
worker.sql-asm-debug.js   10,867,832  10,899,179  bytes
sql-asm.js                 2,383,575   2,389,992  bytes
worker.sql-asm.js          2,386,058   2,392,475  bytes
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@lovasoa

lovasoa commented Oct 26, 2020

Copy link
Copy Markdown
Member

If we want to be able to merge this without breaking backwards compatibility, we can first add -s LEGACY_VM_SUPPORT=1 and -s ALLOW_MEMORY_GROWTH=1 into the asmjs version, and add a deprecation warning to the memory growth version.

On the next major version bump, we can delete the memory growth version.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@kaizhu256
deprecate sql-asm-memory-growth.js in favor of sql-asm.js, which now …
21c9bee
…includes memory-growth and ie11 support.
@kaizhu256

kaizhu256 commented Oct 26, 2020

Copy link
Copy Markdown
Member Author

k. would updated patch with following be ok?

image

image

lovasoa reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@kaizhu256
remove build/test references to sql-asm-memory-growth.js, since its n…
47dd311
…ow essentially a copy of sql-asm.js
@kaizhu256

kaizhu256 commented Oct 27, 2020

Copy link
Copy Markdown
Member Author

latest-patch removes redundant npm-tests on sql-asm-memory-growth.js, since its now essentially a carbon-copy of sql-asm.js (plus a deprecation-warning header).

this simplifies sunsetting sql-asm-memory-growth.js in some future-release by simply deleting the following lines in /Makefile:

--- a/Makefile
+++ b/Makefile
@@ -101,9 +101,6 @@ dist/sql-wasm.js: $(BITCODE_FILES) $(OUTPUT_WRAPPER_FILES) $(SOURCE_API_FILES) $
        mv $@ out/tmp-raw.js
        cat src/shell-pre.js out/tmp-raw.js src/shell-post.js > $@
        rm out/tmp-raw.js
-       # TODO - remove code below in future releases to sunset sql-asm-memory-growth.js
-       printf 'console.error(\n    "\\n\\nDEPRECATION WARNING.\\n"\n    + "sql-asm-memory-growth.js will be removed in future releases.\\n"\n    + "Use sql-asm.js instead, which now includes memory-growth support.\\n\\n"\n);\n' > dist/sql-asm-memory-growth.js
-       cat $@ >> dist/sql-asm-memory-growth.js

 # Web worker API
 .PHONY: worker

if you still want npm-tests on sql-asm-memory-growth.js i can revert this patch

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@kaizhu256 kaizhu256 mentioned this pull request Nov 19, 2020
[RFC] dist/sql-asm-memory-growth-debug.js build target support #330
Closed

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@kaizhu256 @lovasoa
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.