27514


RDS for PostgreSQLから、自宅サーバのPostgreSQLへデータの転送をしようと考えています。  
embulk previewは問題無く通るのですが、runしてみると例外がでてエラーとなってしまいます。  
原因が分かりますでしょうか？

Embulk v0.8.9  
PostgreSQL 9.5.2  
CentOS 7

in側テーブル

[code]

    CREATE TABLE public.tt_test_fav
    (
      status_id bigint NOT NULL,
      id bigint NOT NULL,
      tp smallint NOT NULL DEFAULT 0,
      user_id bigint,
      user_rank smallint NOT NULL DEFAULT 0,
      CONSTRAINT tt_test_fav_pkey PRIMARY KEY (status_id, id, tp)
    );
    
[/code]

out側テーブル

[code]

    CREATE TABLE public.tt_test_fav
    (
      status_id bigint NOT NULL,
      id bigint NOT NULL,
      tp smallint NOT NULL DEFAULT 0,
      user_id bigint,
      CONSTRAINT tt_test_fav_pkey PRIMARY KEY (status_id, id, tp)
    );
    
[/code]

最後のカラム user_rank を除いて転送したい

yaml

[code]

    in:
      type: postgresql
      host: hoge.rds.amazonaws.com
      port: 5432
      user: hoge
      password: hoge
      database: hoge
      table: tt_test_fav
      select: status_id, id, tp, user_id
      column_options:
        status_id: {value_type: long}
        id: {value_type: long}
        tp: {value_type: long}
        user_id: {value_type: long}
    
    out:
      type: postgresql
      host: localhost
      port: 5432
      user: hoge
      password: hoge
      database: hoge
      table: tt_test_fav
      mode: merge
      merge_keys: status_id, id, tp
      column_options:
        status_id: {value_type: long}
        id: {value_type: long}
        tp: {value_type: long}
        user_id: {value_type: long}
    
[/code]

コンソールに出力されるエラー

[code]

    2016-07-09 21:19:35.222 +0900: Embulk v0.8.9
    2016-07-09 21:19:36.750 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-postgresql (0.7.2)
    2016-07-09 21:19:36.963 +0900 [INFO] (0001:transaction): Loaded plugin embulk-output-postgresql (0.6.1)
    2016-07-09 21:19:38.043 +0900 [INFO] (0001:transaction): SQL: SET search_path TO "public"
    2016-07-09 21:19:38.447 +0900 [INFO] (0001:transaction): SQL: SELECT status_id, id, tp, user_id FROM "tt_test_fav"
    2016-07-09 21:19:38.861 +0900 [INFO] (0001:transaction): Using local thread executor with max_threads=40 / output tasks 20 = input tasks 1 * 20
    org.embulk.exec.PartialExecutionException: org.embulk.config.ConfigException: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.ArrayList out of VALUE_STRING token
     at [Source: N/A; line: -1, column: -1]
        at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(org/embulk/exec/BulkLoader.java:363)
        at org.embulk.exec.BulkLoader.doRun(org/embulk/exec/BulkLoader.java:572)
        at org.embulk.exec.BulkLoader.access$000(org/embulk/exec/BulkLoader.java:33)
        at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:374)
        at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:370)
        at org.embulk.spi.Exec.doWith(org/embulk/spi/Exec.java:25)
        at org.embulk.exec.BulkLoader.run(org/embulk/exec/BulkLoader.java:370)
        at org.embulk.EmbulkEmbed.run(org/embulk/EmbulkEmbed.java:180)
        at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
        at RUBY.run(/home/osa/.embulk/bin/embulk!/embulk/runner.rb:84)
        at RUBY.run(/home/osa/.embulk/bin/embulk!/embulk/command/embulk_run.rb:306)
        at RUBY.<top>(/home/osa/.embulk/bin/embulk!/embulk/command/embulk_main.rb:2)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:937)
        at RUBY.(root)(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1)
        at home.osa.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.<top>(file:/home/osa/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:51)
        at java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)
        at org.embulk.cli.Main.main(org/embulk/cli/Main.java:23)
        Suppressed: java.lang.NullPointerException
            at org.embulk.exec.BulkLoader.doCleanup(BulkLoader.java:479)
            at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:410)
            at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:406)
            at org.embulk.spi.Exec.doWith(Exec.java:25)
            at org.embulk.exec.BulkLoader.cleanup(BulkLoader.java:406)
            at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:184)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:452)
            at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:313)
            at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:45)
            at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
            at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
            at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
            at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
            at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:140)
            at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:126)
            at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:273)
            at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
            at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
            at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:414)
            at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:348)
            at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
            at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)
            at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:210)
            at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:196)
            at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
            at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
            at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
            at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
            at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
            at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
            at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
            at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
            at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
            at org.jruby.Ruby.runInterpreter(Ruby.java:833)
            at org.jruby.Ruby.loadFile(Ruby.java:2905)
            at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:235)
            at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
            at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:900)
            at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:541)
            at org.jruby.runtime.load.LoadService.requireCommon(LoadService.java:426)
            at org.jruby.runtime.load.LoadService.require(LoadService.java:392)
            at org.jruby.RubyKernel.requireCommon(RubyKernel.java:944)
            at org.jruby.RubyKernel.require19(RubyKernel.java:937)
            at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(RubyKernel$INVOKER$s$1$0$require19.gen)
            at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:380)
            at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
            at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
            at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
            at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
            at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
            at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)
            at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:210)
            at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:196)
            at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
            at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
            at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
            at home.osa.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.invokeOther66:require(file:/home/osa/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb)
            at home.osa.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.RUBY$script(file:/home/osa/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:51)
            at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
            at org.jruby.ir.Compiler$1.load(Compiler.java:111)
            at org.jruby.Ruby.runScript(Ruby.java:817)
            at org.jruby.Ruby.runScript(Ruby.java:809)
            at org.jruby.Ruby.runNormally(Ruby.java:747)
            at org.jruby.Ruby.runFromMain(Ruby.java:569)
            at org.jruby.Main.doRunFromMain(Main.java:415)
            at org.jruby.Main.internalRun(Main.java:310)
            at org.jruby.Main.run(Main.java:239)
            at org.jruby.Main.main(Main.java:201)
            at org.embulk.cli.Main.main(Main.java:23)
    Caused by: org.embulk.config.ConfigException: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.ArrayList out of VALUE_STRING token
     at [Source: N/A; line: -1, column: -1]
        at org.embulk.config.ModelManager.readObjectWithConfigSerDe(ModelManager.java:75)
        at org.embulk.config.DataSourceImpl.loadConfig(DataSourceImpl.java:220)
        at org.embulk.output.jdbc.AbstractJdbcOutputPlugin.transaction(AbstractJdbcOutputPlugin.java:321)
        at org.embulk.exec.BulkLoader$4$1$1.transaction(BulkLoader.java:529)
        at org.embulk.exec.LocalExecutorPlugin.transaction(LocalExecutorPlugin.java:54)
        at org.embulk.exec.BulkLoader$4$1.run(BulkLoader.java:524)
        at org.embulk.spi.util.Filters$RecursiveControl.transaction(Filters.java:96)
        at org.embulk.spi.util.Filters.transaction(Filters.java:49)
        at org.embulk.exec.BulkLoader$4.run(BulkLoader.java:518)
        at org.embulk.input.jdbc.AbstractJdbcInputPlugin.transaction(AbstractJdbcInputPlugin.java:163)
        at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:513)
        at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:33)
        at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:374)
        at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:370)
        at org.embulk.spi.Exec.doWith(Exec.java:25)
        at org.embulk.exec.BulkLoader.run(BulkLoader.java:370)
        at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:180)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:452)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:313)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:45)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:140)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:126)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:273)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
        at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:414)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:348)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:210)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:196)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
        at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
        at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
        at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
        at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
        at org.jruby.Ruby.runInterpreter(Ruby.java:833)
        at org.jruby.Ruby.loadFile(Ruby.java:2905)
        at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:235)
        at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
        at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:900)
        at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:541)
        at org.jruby.runtime.load.LoadService.requireCommon(LoadService.java:426)
        at org.jruby.runtime.load.LoadService.require(LoadService.java:392)
        at org.jruby.RubyKernel.requireCommon(RubyKernel.java:944)
        at org.jruby.RubyKernel.require19(RubyKernel.java:937)
        at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(RubyKernel$INVOKER$s$1$0$require19.gen)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:380)
        at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:210)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:196)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at home.osa.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.invokeOther66:require(file:/home/osa/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb)
        at home.osa.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.RUBY$script(file:/home/osa/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:51)
        at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
        at org.jruby.ir.Compiler$1.load(Compiler.java:111)
        at org.jruby.Ruby.runScript(Ruby.java:817)
        at org.jruby.Ruby.runScript(Ruby.java:809)
        at org.jruby.Ruby.runNormally(Ruby.java:747)
        at org.jruby.Ruby.runFromMain(Ruby.java:569)
        at org.jruby.Main.doRunFromMain(Main.java:415)
        at org.jruby.Main.internalRun(Main.java:310)
        at org.jruby.Main.run(Main.java:239)
        at org.jruby.Main.main(Main.java:201)
        at org.embulk.cli.Main.main(Main.java:23)
    Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.ArrayList out of VALUE_STRING token
     at [Source: N/A; line: -1, column: -1]
        at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148)
        at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:835)
        at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:831)
        at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.handleNonArray(StringCollectionDeserializer.java:240)
        at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:171)
        at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:161)
        at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:19)
        at com.fasterxml.jackson.datatype.guava.deser.GuavaOptionalDeserializer.deserialize(GuavaOptionalDeserializer.java:98)
        at com.fasterxml.jackson.datatype.guava.deser.GuavaOptionalDeserializer.deserialize(GuavaOptionalDeserializer.java:18)
        at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3534)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1894)
        at org.embulk.config.TaskSerDe$TaskDeserializer.deserialize(TaskSerDe.java:161)
        at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3534)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1870)
        at org.embulk.config.ModelManager.readObjectWithConfigSerDe(ModelManager.java:72)
        ... 83 more
    
    Error: org.embulk.config.ConfigException: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.ArrayList out of VALUE_STRING token
     at [Source: N/A; line: -1, column: -1]
    
[/code]


CC BY-SA 3.0
2016-07-09T12:46:00.357
27479
2016-07-11T00:18:27.603
2016-07-09T14:50:23.310
5337
2605
question
java
Embulkでorg.embulk.config.ConfigExceptionの原因が分からない。


<https://github.com/embulk/embulk-output-jdbc/tree/master/embulk-output-
postgresql>

> merge_keys: key column names for merging records in merge mode (string
> array, required in merge mode)

となっているのでmerge_keyは配列で書きます。

[code]

    merge_keys:
      - status_id
      - id
      - tp
    
[/code]

あるいは`merge_keys: [ status_id, id, tp]`です。

こちらで試した設定の例です。

[code]

    in:
      type: postgresql
      host: 127.0.0.1
      port: 5432
      user: user
      password: pass
      database: embulk_test
      table: tt_test_fav
      select: status_id, id, tp, user_id
    
    out:
      type: postgresql
      host: 127.0.0.1
      port: 5432
      user: user
      password: pass
      database: embulk_test
      table: tt_test_fav_out
      mode: merge
      merge_keys:
        - status_id
        - id
        - tp
    
[/code]


CC BY-SA 3.0
2016-07-11T00:12:03.913
27514
2016-07-11T00:18:27.603
2016-07-11T00:18:27.603
17128
17128
27479
answer