Golang + postgres storing gob data
I'm trying to store encoded data using encoding/gob from Golang into Postgres. I'm using Gorm as well.
First, sending form data using
if err := json.NewDecoder(r.Body).Decode(model); err != nil
http.Error(w, err.Error(), http.StatusBadRequest)
return
Currently client_encoding is set to UTF8 in the postgres database. Here's what I'm using to encode:
type payload struct
Key byte
Nonce *[nonceLength]byte
Message byte
// Encrypt message
p.Message = secretbox.Seal(p.Message, plaintext, p.Nonce, key) // key set prior to this
buf := &bytes.Buffer
if err := gob.NewEncoder(buf).Encode(p); err != nil
return nil, err
return buf.Bytes(), nil
Then I store string(buf.Bytes()) which is stored in the database column which is currently a string type. Now I'm a novice with encoding, and I think gob just has a different encoding for my database. I'm receiving this error in console:
(pq: invalid byte sequence for encoding "UTF8": 0xff)
I've been following this gist for encryption/decryption:
https://gist.github.com/fuzzyami/f3a7231037166117a6fef9607960aee7
From what I've read, I shouldn't be encoding structs into the db, in this case p, unless using gob. Correct me if I'm wrong with that (can't find the resource at the moment where I found this).
Is anyone able to point me in the right direction for storing this data in Postgres which is decrypted later? I'm clearly not understanding the encoding process, and not entirely sure where to start out here with reading resources, so any help is appreciated!
postgresql go gob
add a comment |
I'm trying to store encoded data using encoding/gob from Golang into Postgres. I'm using Gorm as well.
First, sending form data using
if err := json.NewDecoder(r.Body).Decode(model); err != nil
http.Error(w, err.Error(), http.StatusBadRequest)
return
Currently client_encoding is set to UTF8 in the postgres database. Here's what I'm using to encode:
type payload struct
Key byte
Nonce *[nonceLength]byte
Message byte
// Encrypt message
p.Message = secretbox.Seal(p.Message, plaintext, p.Nonce, key) // key set prior to this
buf := &bytes.Buffer
if err := gob.NewEncoder(buf).Encode(p); err != nil
return nil, err
return buf.Bytes(), nil
Then I store string(buf.Bytes()) which is stored in the database column which is currently a string type. Now I'm a novice with encoding, and I think gob just has a different encoding for my database. I'm receiving this error in console:
(pq: invalid byte sequence for encoding "UTF8": 0xff)
I've been following this gist for encryption/decryption:
https://gist.github.com/fuzzyami/f3a7231037166117a6fef9607960aee7
From what I've read, I shouldn't be encoding structs into the db, in this case p, unless using gob. Correct me if I'm wrong with that (can't find the resource at the moment where I found this).
Is anyone able to point me in the right direction for storing this data in Postgres which is decrypted later? I'm clearly not understanding the encoding process, and not entirely sure where to start out here with reading resources, so any help is appreciated!
postgresql go gob
add a comment |
I'm trying to store encoded data using encoding/gob from Golang into Postgres. I'm using Gorm as well.
First, sending form data using
if err := json.NewDecoder(r.Body).Decode(model); err != nil
http.Error(w, err.Error(), http.StatusBadRequest)
return
Currently client_encoding is set to UTF8 in the postgres database. Here's what I'm using to encode:
type payload struct
Key byte
Nonce *[nonceLength]byte
Message byte
// Encrypt message
p.Message = secretbox.Seal(p.Message, plaintext, p.Nonce, key) // key set prior to this
buf := &bytes.Buffer
if err := gob.NewEncoder(buf).Encode(p); err != nil
return nil, err
return buf.Bytes(), nil
Then I store string(buf.Bytes()) which is stored in the database column which is currently a string type. Now I'm a novice with encoding, and I think gob just has a different encoding for my database. I'm receiving this error in console:
(pq: invalid byte sequence for encoding "UTF8": 0xff)
I've been following this gist for encryption/decryption:
https://gist.github.com/fuzzyami/f3a7231037166117a6fef9607960aee7
From what I've read, I shouldn't be encoding structs into the db, in this case p, unless using gob. Correct me if I'm wrong with that (can't find the resource at the moment where I found this).
Is anyone able to point me in the right direction for storing this data in Postgres which is decrypted later? I'm clearly not understanding the encoding process, and not entirely sure where to start out here with reading resources, so any help is appreciated!
postgresql go gob
I'm trying to store encoded data using encoding/gob from Golang into Postgres. I'm using Gorm as well.
First, sending form data using
if err := json.NewDecoder(r.Body).Decode(model); err != nil
http.Error(w, err.Error(), http.StatusBadRequest)
return
Currently client_encoding is set to UTF8 in the postgres database. Here's what I'm using to encode:
type payload struct
Key byte
Nonce *[nonceLength]byte
Message byte
// Encrypt message
p.Message = secretbox.Seal(p.Message, plaintext, p.Nonce, key) // key set prior to this
buf := &bytes.Buffer
if err := gob.NewEncoder(buf).Encode(p); err != nil
return nil, err
return buf.Bytes(), nil
Then I store string(buf.Bytes()) which is stored in the database column which is currently a string type. Now I'm a novice with encoding, and I think gob just has a different encoding for my database. I'm receiving this error in console:
(pq: invalid byte sequence for encoding "UTF8": 0xff)
I've been following this gist for encryption/decryption:
https://gist.github.com/fuzzyami/f3a7231037166117a6fef9607960aee7
From what I've read, I shouldn't be encoding structs into the db, in this case p, unless using gob. Correct me if I'm wrong with that (can't find the resource at the moment where I found this).
Is anyone able to point me in the right direction for storing this data in Postgres which is decrypted later? I'm clearly not understanding the encoding process, and not entirely sure where to start out here with reading resources, so any help is appreciated!
postgresql go gob
postgresql go gob
edited Nov 11 '18 at 18:43
Greg Miller
asked Nov 11 '18 at 17:58
Greg MillerGreg Miller
147110
147110
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
Use a bytea column in postgresql to store a byte, and skip the conversion to string.
I've just tried to implement this, but when coding the body into the struct I getjson: cannot unmarshal string into Go struct field Table.Column of type pq.ByteaArray. Any thoughts on getting around this? Seems like this can get a bit messy.
– Greg Miller
Nov 11 '18 at 18:35
That's coming from code that isn't in your question, and you're doing something very wrong.
– hobbs
Nov 11 '18 at 18:38
Just passing form data and decoding into a struct. I'll update the question now.
– Greg Miller
Nov 11 '18 at 18:42
add a comment |
Took a look at https://golang.org/src/encoding/base64/example_test.go
Was able to use
return base64.StdEncoding.EncodeToString(buf.Bytes()), nil
Which successfully stored in the database.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53251572%2fgolang-postgres-storing-gob-data%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Use a bytea column in postgresql to store a byte, and skip the conversion to string.
I've just tried to implement this, but when coding the body into the struct I getjson: cannot unmarshal string into Go struct field Table.Column of type pq.ByteaArray. Any thoughts on getting around this? Seems like this can get a bit messy.
– Greg Miller
Nov 11 '18 at 18:35
That's coming from code that isn't in your question, and you're doing something very wrong.
– hobbs
Nov 11 '18 at 18:38
Just passing form data and decoding into a struct. I'll update the question now.
– Greg Miller
Nov 11 '18 at 18:42
add a comment |
Use a bytea column in postgresql to store a byte, and skip the conversion to string.
I've just tried to implement this, but when coding the body into the struct I getjson: cannot unmarshal string into Go struct field Table.Column of type pq.ByteaArray. Any thoughts on getting around this? Seems like this can get a bit messy.
– Greg Miller
Nov 11 '18 at 18:35
That's coming from code that isn't in your question, and you're doing something very wrong.
– hobbs
Nov 11 '18 at 18:38
Just passing form data and decoding into a struct. I'll update the question now.
– Greg Miller
Nov 11 '18 at 18:42
add a comment |
Use a bytea column in postgresql to store a byte, and skip the conversion to string.
Use a bytea column in postgresql to store a byte, and skip the conversion to string.
answered Nov 11 '18 at 18:01
hobbshobbs
142k14150236
142k14150236
I've just tried to implement this, but when coding the body into the struct I getjson: cannot unmarshal string into Go struct field Table.Column of type pq.ByteaArray. Any thoughts on getting around this? Seems like this can get a bit messy.
– Greg Miller
Nov 11 '18 at 18:35
That's coming from code that isn't in your question, and you're doing something very wrong.
– hobbs
Nov 11 '18 at 18:38
Just passing form data and decoding into a struct. I'll update the question now.
– Greg Miller
Nov 11 '18 at 18:42
add a comment |
I've just tried to implement this, but when coding the body into the struct I getjson: cannot unmarshal string into Go struct field Table.Column of type pq.ByteaArray. Any thoughts on getting around this? Seems like this can get a bit messy.
– Greg Miller
Nov 11 '18 at 18:35
That's coming from code that isn't in your question, and you're doing something very wrong.
– hobbs
Nov 11 '18 at 18:38
Just passing form data and decoding into a struct. I'll update the question now.
– Greg Miller
Nov 11 '18 at 18:42
I've just tried to implement this, but when coding the body into the struct I get
json: cannot unmarshal string into Go struct field Table.Column of type pq.ByteaArray. Any thoughts on getting around this? Seems like this can get a bit messy.– Greg Miller
Nov 11 '18 at 18:35
I've just tried to implement this, but when coding the body into the struct I get
json: cannot unmarshal string into Go struct field Table.Column of type pq.ByteaArray. Any thoughts on getting around this? Seems like this can get a bit messy.– Greg Miller
Nov 11 '18 at 18:35
That's coming from code that isn't in your question, and you're doing something very wrong.
– hobbs
Nov 11 '18 at 18:38
That's coming from code that isn't in your question, and you're doing something very wrong.
– hobbs
Nov 11 '18 at 18:38
Just passing form data and decoding into a struct. I'll update the question now.
– Greg Miller
Nov 11 '18 at 18:42
Just passing form data and decoding into a struct. I'll update the question now.
– Greg Miller
Nov 11 '18 at 18:42
add a comment |
Took a look at https://golang.org/src/encoding/base64/example_test.go
Was able to use
return base64.StdEncoding.EncodeToString(buf.Bytes()), nil
Which successfully stored in the database.
add a comment |
Took a look at https://golang.org/src/encoding/base64/example_test.go
Was able to use
return base64.StdEncoding.EncodeToString(buf.Bytes()), nil
Which successfully stored in the database.
add a comment |
Took a look at https://golang.org/src/encoding/base64/example_test.go
Was able to use
return base64.StdEncoding.EncodeToString(buf.Bytes()), nil
Which successfully stored in the database.
Took a look at https://golang.org/src/encoding/base64/example_test.go
Was able to use
return base64.StdEncoding.EncodeToString(buf.Bytes()), nil
Which successfully stored in the database.
answered Nov 11 '18 at 20:58
Greg MillerGreg Miller
147110
147110
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53251572%2fgolang-postgres-storing-gob-data%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown