GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Improve peformances put/get rows using pandas.DataFrame by dangtrungtin · Pull Request #19 · griddb/python_client

Conversation

@dangtrungtin

@dangtrungtin dangtrungtin commented Sep 9, 2020

Copy link
Copy Markdown
Contributor
  • Add function : void Container.put_rows(pandas.DataFrame input). In python layer, I convert input from pandas.DataFrame to numpy.array because NumPy support C-API. In C++ layer, I use API from NumPy to put data into GridDB. Compare with using Container.multi_puts(input : list[list]) to put large data with LONG type, the time to run reduce about 11% and memory using reduce 20%. With String type, the time to run reduce about 10% and memory using reduce about 8%.

  • Add function : pandas.DataFrame RowSet.fetch_rows(). In C++ layer, it uses Iterable object (RowList.h/cpp) to wrap output data. In python layer, I convert data from Iterable object to pandas.DataFrame. Compare with using RowSet.next() to query large data with LONG type, the time to run reduce 17% and memory using are the same. When query large data with STRING type, the time to run reduce 11%.

  • Reduce call function to check NULL field: when get data, for each row field, Python Client is using gsGetRowFieldNull() then gsGetRowFieldAsXXX(). I change to use gsGetRowFieldAsXXX(), then if data is empty or null then I use gsGetRowFieldNull() to check whether field is null.

  • There is a note for Container.put_rows(). To create DataFrame, we use: "frame = pandas.DataFrame(data)" with "data" is list. However, when list has None value, Pandas library will automatic change value, for example None value to NaN value. To prevent this, in python code should use "frame = pandas.DataFrame(data, dtype=object)".

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@dangtrungtin
Improve performance put/get rows using pandas.DataFrame
37297ad
- Add function: void Container.put_rows(input: pandas.DataFrame)
- Add function: pandas.DataFrame RowSet.fetch_rows()
- Reduce call function to check NULL field
@knonomura

knonomura commented Sep 10, 2020

Copy link
Copy Markdown
Member

Oh, that's great !
I'll try to use it.

I have a question.
How much data did you use?

Thanks.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@dangtrungtin

dangtrungtin commented Sep 10, 2020

Copy link
Copy Markdown
Contributor Author

With long type, I put 1000 rows x 10000 fields. With string type, I use 1000 rows x 7552 fields.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@knonomura

knonomura commented Sep 11, 2020

Copy link
Copy Markdown
Member

Thank you for your information.
I understand.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@knonomura

knonomura commented Sep 15, 2020

Copy link
Copy Markdown
Member

I have a request.
Could you please add a sample for new function ?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@dangtrungtin
Add sample for put/fetch rows with DataFrame.
fee8818
- PutRowsWithDataFrame.py : sample for put rows.
- FetchRowsWithDataFrame.py : sample for fetch rows.
@dangtrungtin

dangtrungtin commented Sep 16, 2020

Copy link
Copy Markdown
Contributor Author

I added 2 samples:

  • PutRowsWithDataFrame.py : sample for put rows.
  • FetchRowsWithDataFrame.py : sample for fetch rows.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@knonomura

knonomura commented Sep 16, 2020

Copy link
Copy Markdown
Member

Thank you for your samples.
I'll check them.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@knonomura

knonomura commented Sep 21, 2020

Copy link
Copy Markdown
Member

I guess this pull request is very useful.
So I merge it.
Thank you.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@knonomura
knonomura merged commit eda9482 into griddb:master Sep 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@dangtrungtin @knonomura
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.